[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:29:48 dist-test-slave-87bt krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:29:48 dist-test-slave-87bt krb5kdc[395](info): set up 2 sockets
Sep 26 21:29:48 dist-test-slave-87bt krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 21:29:49.519584 388 mini_kdc.cc:121] Time spent starting KDC: real 2.282s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 21:29:49.688560 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.168s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 21:29:49.857805 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.169s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 21:29:50.091614 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.234s user 0.002s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/test-user.keytab.
W20260926 21:29:50.781495 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.690s user 0.000s sys 0.004s
Sep 26 21:29:50 dist-test-slave-87bt krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:29:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:29:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:29:50.862149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36871
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:51.184566 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:51.184885 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:51.185031 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:51.192474 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:29:51.192595 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:51.192680 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:51.192770 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:29:51.192847 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:29:51.203372 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:51.204739 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:51.206190 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:51.218204 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:51.218148 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:51.218245 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:51.219307 408 server_base.cc:1034] running on GCE node
I20260926 21:29:51.220104 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:51.221514 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:51.222723 408 hybrid_clock.cc:648] HybridClock initialized: now 1790458191222637 us; error 85 us; skew 500 ppm
I20260926 21:29:51.223067 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:51 dist-test-slave-87bt krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:51.226035 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:29:51.227526 408 webserver.cc:460] Webserver started at http://127.0.97.62:45059/ using document root <none> and password file <none>
I20260926 21:29:51.228185 408 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:51.228282 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:51.228525 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:51.231503 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "cf879568e06f4d0a88b43c3218028b0c"
format_stamp: "Formatted at 2026-09-26 21:29:51 on dist-test-slave-87bt"
server_key: "199ee0525165ccefcd8a06fdccdee368"
server_key_iv: "761d788ee538fde2ee248528c726d7b5"
server_key_version: "encryptionkey@0"
I20260926 21:29:51.232200 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "cf879568e06f4d0a88b43c3218028b0c"
format_stamp: "Formatted at 2026-09-26 21:29:51 on dist-test-slave-87bt"
server_key: "199ee0525165ccefcd8a06fdccdee368"
server_key_iv: "761d788ee538fde2ee248528c726d7b5"
server_key_version: "encryptionkey@0"
I20260926 21:29:51.237576 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:29:51.240953 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:51.242235 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:29:51.242364 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "cf879568e06f4d0a88b43c3218028b0c"
format_stamp: "Formatted at 2026-09-26 21:29:51 on dist-test-slave-87bt"
server_key: "199ee0525165ccefcd8a06fdccdee368"
server_key_iv: "761d788ee538fde2ee248528c726d7b5"
server_key_version: "encryptionkey@0"
I20260926 21:29:51.242575 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:51.273073 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:51.276808 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:51.277042 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:51.296018 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36871
I20260926 21:29:51.296048 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36871 every 8 connection(s)
I20260926 21:29:51.297359 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:29:51.300937 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:29:51.308458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 408
I20260926 21:29:51.308768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:29:51.311108 388 external_mini_cluster.cc:1428] Setting key 33b4ca787b4fe6c5e7a02cd7e6f4c942
I20260926 21:29:51.312512 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c: Bootstrap starting.
I20260926 21:29:51.316149 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c: Neither blocks nor log segments found. Creating new log.
I20260926 21:29:51.317205 475 log.cc:824] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c: Log is configured to *not* fsync() on all Append() calls
I20260926 21:29:51.319758 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c: No bootstrap required, opened a new log
I20260926 21:29:51.328052 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf879568e06f4d0a88b43c3218028b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } }
I20260926 21:29:51.328413 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:29:51.328536 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf879568e06f4d0a88b43c3218028b0c, State: Initialized, Role: FOLLOWER
I20260926 21:29:51.329380 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf879568e06f4d0a88b43c3218028b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } }
I20260926 21:29:51.329619 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:29:51.329751 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:29:51.329921 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:29:51.332077 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf879568e06f4d0a88b43c3218028b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } }
I20260926 21:29:51.332584 475 leader_election.cc:302] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf879568e06f4d0a88b43c3218028b0c; no voters:
I20260926 21:29:51.333298 475 leader_election.cc:288] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:29:51.333425 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:29:51.334811 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [term 1 LEADER]: Becoming Leader. State: Replica: cf879568e06f4d0a88b43c3218028b0c, State: Running, Role: LEADER
I20260926 21:29:51.335424 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf879568e06f4d0a88b43c3218028b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } }
I20260926 21:29:51.335810 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:29:51.338487 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf879568e06f4d0a88b43c3218028b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf879568e06f4d0a88b43c3218028b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } } }
I20260926 21:29:51.339380 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [sys.catalog]: This master's current role is: LEADER
Sep 26 21:29:51 dist-test-slave-87bt krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:29:51.343751 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf879568e06f4d0a88b43c3218028b0c. Latest consensus state: current_term: 1 leader_uuid: "cf879568e06f4d0a88b43c3218028b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf879568e06f4d0a88b43c3218028b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } } }
I20260926 21:29:51.344674 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c [sys.catalog]: This master's current role is: LEADER
I20260926 21:29:51.347656 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:29:51.353641 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:29:51.358351 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:51.315491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.0.97.62:36871)
0926 21:29:51.316389 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:51.316407 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:51.317652 (+ 1245us) server_negotiation.cc:407] Connection header received
0926 21:29:51.327555 (+ 9903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:51.327576 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:51.327940 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:51.328320 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:51.331792 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:51.333265 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:51.335629 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:51.336413 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:51.336553 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:51.345536 (+ 8983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:51.345551 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:51.345581 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:51.345648 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:51.350682 (+ 5034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:51.351781 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:51.351789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:51.351799 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:51.352004 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:51.352617 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:51.352621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:51.352624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:51.353280 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:51.356248 (+ 2968us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:51.356532 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 21:29:51.356988 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":649,"thread_start_us":408,"threads_started":1}
I20260926 21:29:51.366024 486 catalog_manager.cc:1347] Generated new cluster ID: 0f036df7b2b144dc8e206593cddc6e32
I20260926 21:29:51.366151 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:29:51.384642 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:29:51.385865 486 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:29:51.400107 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf879568e06f4d0a88b43c3218028b0c: Generated new TSK 0
I20260926 21:29:51.401136 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:29:51.454864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:51.795027 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:51.795317 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:51.795414 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:51.803285 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:51.803428 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:51.803627 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:29:51.813928 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:51.815248 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:51.816606 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:51.829957 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:51.829916 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:51.830013 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:51.831041 501 server_base.cc:1034] running on GCE node
I20260926 21:29:51.831704 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:51.832697 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:51.833958 501 hybrid_clock.cc:648] HybridClock initialized: now 1790458191833904 us; error 45 us; skew 500 ppm
I20260926 21:29:51.834306 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:51 dist-test-slave-87bt krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:51.837595 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:29:51.839042 501 webserver.cc:460] Webserver started at http://127.0.97.1:36477/ using document root <none> and password file <none>
I20260926 21:29:51.839694 501 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:51.839814 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:51.840122 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:51.843020 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "82f40bf6c223408fb5ac41293e6e6f3c"
format_stamp: "Formatted at 2026-09-26 21:29:51 on dist-test-slave-87bt"
server_key: "700f76e210fab336f7629c4e14f58daf"
server_key_iv: "51b1c7876c18aec4329451dac0e68b9c"
server_key_version: "encryptionkey@0"
I20260926 21:29:51.843756 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "82f40bf6c223408fb5ac41293e6e6f3c"
format_stamp: "Formatted at 2026-09-26 21:29:51 on dist-test-slave-87bt"
server_key: "700f76e210fab336f7629c4e14f58daf"
server_key_iv: "51b1c7876c18aec4329451dac0e68b9c"
server_key_version: "encryptionkey@0"
I20260926 21:29:51.849000 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 21:29:51.852473 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:51.853775 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:29:51.853930 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "82f40bf6c223408fb5ac41293e6e6f3c"
format_stamp: "Formatted at 2026-09-26 21:29:51 on dist-test-slave-87bt"
server_key: "700f76e210fab336f7629c4e14f58daf"
server_key_iv: "51b1c7876c18aec4329451dac0e68b9c"
server_key_version: "encryptionkey@0"
I20260926 21:29:51.854126 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:51.888510 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:51.892885 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:51.893173 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:51.894109 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:29:51.895525 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:29:51.895601 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:51.895752 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:29:51.895813 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:51.929396 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37867
I20260926 21:29:51.929441 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37867 every 8 connection(s)
I20260926 21:29:51.930641 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:29:51.931782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 501
I20260926 21:29:51.931993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:29:51.932760 388 external_mini_cluster.cc:1428] Setting key 5a255cc83ad0991cdd48b6643edfa785
Sep 26 21:29:51 dist-test-slave-87bt krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:29:51.951511 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:51.933379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37499 (local address 127.0.97.62:36871)
0926 21:29:51.934129 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:51.934135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:51.937476 (+ 3341us) server_negotiation.cc:407] Connection header received
0926 21:29:51.937529 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:51.937533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:51.937592 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:51.937711 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:51.939201 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:51.939918 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:51.941446 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:51.941829 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:51.941960 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:51.945318 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:51.945323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:51.945326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:51.945357 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:51.948046 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:51.948793 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:51.948797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:51.948799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:51.948907 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:51.949479 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:51.949482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:51.949484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:51.949804 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:51.949912 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:51.950713 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 21:29:51.950969 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":504,"threads_started":1}
I20260926 21:29:51.952090 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:51.934176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.1:37499)
0926 21:29:51.934990 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:51.935036 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:51.937162 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:51.937937 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:51.937945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:51.938504 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:29:51.939039 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:51.939056 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:51.940073 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:51.940082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:51.941291 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:51.941303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:51.942055 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:51.942064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:51.942303 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:51.943004 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:29:51.943030 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:29:51.945069 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:29:51.948193 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:51.948201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:51.948230 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:51.948682 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:51.948994 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:51.948997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:51.948999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:51.949219 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:51.949923 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:29:51.949934 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:29:51.950291 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 21:29:51.950564 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 21:29:51.950946 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":355,"threads_started":1}
I20260926 21:29:51.953797 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36871
I20260926 21:29:51.954131 629 heartbeater.cc:464] Registering TS with master...
I20260926 21:29:51.955063 629 heartbeater.cc:511] Master 127.0.97.62:36871 requested a full tablet report, sending...
I20260926 21:29:51.957654 440 ts_manager.cc:194] Registered new tserver with Master: 82f40bf6c223408fb5ac41293e6e6f3c (127.0.97.1:37867)
I20260926 21:29:51.959295 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:29:51.973462 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:52.318730 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:52.319048 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:52.319214 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:52.326926 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:52.327092 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:52.327270 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:29:52.337698 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:52.339202 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:52.340566 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:52.354547 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:52.354601 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:52.354810 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:52.355834 635 server_base.cc:1034] running on GCE node
I20260926 21:29:52.356523 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:52.357530 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:52.358755 635 hybrid_clock.cc:648] HybridClock initialized: now 1790458192358672 us; error 72 us; skew 500 ppm
I20260926 21:29:52.359071 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:52 dist-test-slave-87bt krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:52.362540 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:29:52.364145 635 webserver.cc:460] Webserver started at http://127.0.97.2:35563/ using document root <none> and password file <none>
I20260926 21:29:52.364910 635 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:52.365037 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:52.365345 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:52.368502 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "104b52e2bbf2436aa7171de187263563"
format_stamp: "Formatted at 2026-09-26 21:29:52 on dist-test-slave-87bt"
server_key: "c4441bb8b6e9bd1c54b02f6e9046c1b1"
server_key_iv: "3468965fc3a55f5c12ae969ef5a47e30"
server_key_version: "encryptionkey@0"
I20260926 21:29:52.369385 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "104b52e2bbf2436aa7171de187263563"
format_stamp: "Formatted at 2026-09-26 21:29:52 on dist-test-slave-87bt"
server_key: "c4441bb8b6e9bd1c54b02f6e9046c1b1"
server_key_iv: "3468965fc3a55f5c12ae969ef5a47e30"
server_key_version: "encryptionkey@0"
I20260926 21:29:52.375123 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:29:52.379632 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:52.381040 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:29:52.381237 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "104b52e2bbf2436aa7171de187263563"
format_stamp: "Formatted at 2026-09-26 21:29:52 on dist-test-slave-87bt"
server_key: "c4441bb8b6e9bd1c54b02f6e9046c1b1"
server_key_iv: "3468965fc3a55f5c12ae969ef5a47e30"
server_key_version: "encryptionkey@0"
I20260926 21:29:52.381445 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:52.400974 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:52.405637 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:52.405923 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:52.406929 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:29:52.408268 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:29:52.408365 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:52.408509 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:29:52.408576 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:52.443521 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45099
I20260926 21:29:52.443560 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45099 every 8 connection(s)
I20260926 21:29:52.444768 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:29:52.450124 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 635
I20260926 21:29:52.450306 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:29:52.450743 388 external_mini_cluster.cc:1428] Setting key ee6e31929cc397367e9a0544ba6ceb9b
Sep 26 21:29:52 dist-test-slave-87bt krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:29:52.467856 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:52.447337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47699 (local address 127.0.97.62:36871)
0926 21:29:52.447537 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:52.447543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:52.448800 (+ 1257us) server_negotiation.cc:407] Connection header received
0926 21:29:52.450705 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:52.450710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:52.450803 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:52.450924 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:52.452826 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:52.453512 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:52.456679 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:52.457100 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:52.457206 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:52.461029 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:52.461037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:52.461045 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:52.461089 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:52.464702 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:52.465390 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:52.465397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:52.465401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:52.465559 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:52.465957 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:52.465963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:52.465968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:52.466348 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:52.466535 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:52.467210 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 21:29:52.467456 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:29:52.468488 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:52.447890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.2:47699)
0926 21:29:52.448630 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:52.448672 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:52.450411 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:52.451219 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:52.451231 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:52.451912 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:29:52.452581 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:52.452611 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:52.453823 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:52.453831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:52.456510 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:52.456528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:52.457326 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:52.457336 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:52.457629 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:52.458301 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:29:52.458328 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:29:52.460789 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:29:52.464838 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:52.464844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:52.464858 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:52.465260 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:52.465669 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:52.465672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:52.465674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:52.465837 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:52.466508 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:29:52.466514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:29:52.466836 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 21:29:52.467051 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 21:29:52.467423 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":319,"threads_started":1}
I20260926 21:29:52.470000 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36871
I20260926 21:29:52.470289 763 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 21:29:52.471130 763 heartbeater.cc:511] Master 127.0.97.62:36871 requested a full tablet report, sending...
I20260926 21:29:52.472661 440 ts_manager.cc:194] Registered new tserver with Master: 104b52e2bbf2436aa7171de187263563 (127.0.97.2:45099)
I20260926 21:29:52.473425 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:29:52.491423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:52.824229 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:52.824513 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:52.824692 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:52.834147 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:52.834342 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:52.834564 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:29:52.847172 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:52.848477 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:52.849848 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:52.863260 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:52.863327 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:52.863260 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:52.864499 768 server_base.cc:1034] running on GCE node
I20260926 21:29:52.865070 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:52.865968 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:52.867161 768 hybrid_clock.cc:648] HybridClock initialized: now 1790458192867103 us; error 41 us; skew 500 ppm
I20260926 21:29:52.867491 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:52 dist-test-slave-87bt krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:52.870780 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:29:52.872731 768 webserver.cc:460] Webserver started at http://127.0.97.3:34187/ using document root <none> and password file <none>
I20260926 21:29:52.873409 768 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:52.873529 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:52.873838 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:52.876785 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "4cee1a58309d4d3e97a0a15ee259ebdb"
format_stamp: "Formatted at 2026-09-26 21:29:52 on dist-test-slave-87bt"
server_key: "d40f862b020bf51017df50ccaed57444"
server_key_iv: "5ec9fd887ff31dfca98e075f225b12ec"
server_key_version: "encryptionkey@0"
I20260926 21:29:52.877575 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4cee1a58309d4d3e97a0a15ee259ebdb"
format_stamp: "Formatted at 2026-09-26 21:29:52 on dist-test-slave-87bt"
server_key: "d40f862b020bf51017df50ccaed57444"
server_key_iv: "5ec9fd887ff31dfca98e075f225b12ec"
server_key_version: "encryptionkey@0"
I20260926 21:29:52.883469 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:29:52.887202 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:52.888514 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:29:52.888690 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "4cee1a58309d4d3e97a0a15ee259ebdb"
format_stamp: "Formatted at 2026-09-26 21:29:52 on dist-test-slave-87bt"
server_key: "d40f862b020bf51017df50ccaed57444"
server_key_iv: "5ec9fd887ff31dfca98e075f225b12ec"
server_key_version: "encryptionkey@0"
I20260926 21:29:52.888880 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:52.918010 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:52.922876 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:52.923195 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:52.924245 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:29:52.925523 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:29:52.925596 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:52.925761 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:29:52.925812 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:52.961380 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33029
I20260926 21:29:52.961423 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33029 every 8 connection(s)
I20260926 21:29:52.962736 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:29:52.963126 629 heartbeater.cc:503] Master 127.0.97.62:36871 was elected leader, sending a full tablet report...
I20260926 21:29:52.969514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 768
I20260926 21:29:52.969693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:29:52.970129 388 external_mini_cluster.cc:1428] Setting key fe25ac012821df3a3df57ae684ff5e6e
Sep 26 21:29:52 dist-test-slave-87bt krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:29:52.982580 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:52.965407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48505 (local address 127.0.97.62:36871)
0926 21:29:52.965564 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:52.965569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:52.966787 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 21:29:52.968548 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:52.968554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:52.968635 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:52.968744 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:52.970775 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:52.971720 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:52.973462 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:52.973994 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:52.974107 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:52.977362 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:52.977367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:52.977371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:52.977403 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:52.979814 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:52.980572 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:52.980576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:52.980578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:52.980710 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:52.981081 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:52.981084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:52.981086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:52.981343 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:52.981449 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:52.982113 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 21:29:52.982274 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:29:52.983532 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:52.965965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.3:48505)
0926 21:29:52.966640 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:52.966677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:52.968297 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:52.968961 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:52.968969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:52.969959 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:29:52.970613 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:52.970632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:52.971853 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:52.971859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:52.973325 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:52.973341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:52.974200 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:52.974206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:52.974394 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:52.975046 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:29:52.975067 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:29:52.977141 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:29:52.979955 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:52.979962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:52.979976 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:52.980394 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:52.980806 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:52.980809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:52.980811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:52.980979 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:52.981446 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:29:52.981452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:29:52.981763 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 21:29:52.981988 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 21:29:52.982376 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":256,"threads_started":1}
I20260926 21:29:52.985172 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36871
I20260926 21:29:52.985574 896 heartbeater.cc:464] Registering TS with master...
I20260926 21:29:52.986300 896 heartbeater.cc:511] Master 127.0.97.62:36871 requested a full tablet report, sending...
I20260926 21:29:52.987711 440 ts_manager.cc:194] Registered new tserver with Master: 4cee1a58309d4d3e97a0a15ee259ebdb (127.0.97.3:33029)
I20260926 21:29:52.988451 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48505
I20260926 21:29:52.998231 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458193, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458193, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:29:53.042546 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:53.026964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.97.1:37867)
0926 21:29:53.027629 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:53.027636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:53.027665 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:29:53.027728 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:53.027735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:53.027956 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:53.028086 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:53.029415 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.030496 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.031747 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.032305 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.032404 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:53.036556 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:53.036570 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:53.036585 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:53.036624 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:53.039989 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.040729 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.040737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.040739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.040866 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.041221 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.041224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.041226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.041543 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:53.041771 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:53.041999 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 21:29:53.042200 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":335,"threads_started":1}
W20260926 21:29:53.043792 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41102
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:29:53.093631 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:53.074163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.0.97.1:37867)
0926 21:29:53.074414 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:53.074445 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:53.074679 (+ 234us) server_negotiation.cc:407] Connection header received
0926 21:29:53.074826 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:53.074830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:53.074925 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:53.075011 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:53.076295 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.077003 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.078150 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.078618 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.078707 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:53.088592 (+ 9885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:53.088598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:53.088602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:53.088645 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:53.091181 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.091830 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.091834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.091836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.091983 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.092394 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.092400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.092403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.092740 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:53.092923 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:53.093204 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 21:29:53.093368 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 21:29:53.095152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 501
I20260926 21:29:53.108338 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:53.096047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.0.97.1:37867)
0926 21:29:53.096223 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:53.096229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:53.096583 (+ 354us) server_negotiation.cc:407] Connection header received
0926 21:29:53.096768 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:53.096771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:53.096823 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:53.096913 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:53.098045 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.099217 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.100470 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.100895 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.101023 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:53.103309 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:53.103315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:53.103318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:53.103351 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:53.105787 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.106476 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.106481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.106485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.106625 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.106993 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.106998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.107000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.107241 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:53.107479 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:53.107761 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 21:29:53.107963 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:29:53.108995 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:29:53.191123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 501
I20260926 21:29:53.207442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 635
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:29:53.222707 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:53.208341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.0.97.2:45099)
0926 21:29:53.208931 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:53.208944 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:53.208969 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:29:53.209035 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:53.209042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:53.209244 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:53.209379 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:53.210582 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.211606 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.212940 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.213479 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.213573 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:53.217485 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:53.217496 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:53.217510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:53.217545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:53.220353 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.220928 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.220935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.220937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.221047 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.221389 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.221392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.221394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.221692 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:53.221901 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:53.222113 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 21:29:53.222300 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":342,"threads_started":1}
I20260926 21:29:53.223896 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:29:53.308763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 635
I20260926 21:29:53.330300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 768
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:29:53.345746 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:53.331274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.0.97.3:33029)
0926 21:29:53.331865 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:53.331872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:53.331899 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:29:53.331953 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:53.331960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:53.332177 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:53.332298 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:53.333437 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.334691 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.335868 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:53.336471 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:53.336568 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:53.340638 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:53.340649 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:53.340664 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:53.340696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:53.343486 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.344020 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.344027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.344029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.344153 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:53.344436 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:53.344439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:53.344441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:53.344738 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:53.344956 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:53.345168 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 21:29:53.345346 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":290,"threads_started":1}
I20260926 21:29:53.347103 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:29:53.427785 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 768
I20260926 21:29:53.443926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 408
I20260926 21:29:53.445065 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:29:53.547242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 408
2026-09-26T21:29:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6364 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[917](info): set up 2 sockets
Sep 26 21:29:53 dist-test-slave-87bt krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 21:29:54.626215 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:29:54 dist-test-slave-87bt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:29:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:29:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:29:54.796190 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37457
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:55.107906 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:55.108189 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:55.108353 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:55.115530 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:29:55.115648 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:55.115724 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:55.115813 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:29:55.115895 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:29:55.125639 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:55.127014 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:55.128299 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:55.139422 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:55.139529 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:55.139509 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:55.140610 930 server_base.cc:1034] running on GCE node
I20260926 21:29:55.141418 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:55.142884 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:55.144090 930 hybrid_clock.cc:648] HybridClock initialized: now 1790458195144027 us; error 52 us; skew 500 ppm
I20260926 21:29:55.144390 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:55 dist-test-slave-87bt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:55.147754 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:29:55.149412 930 webserver.cc:460] Webserver started at http://127.0.97.62:38719/ using document root <none> and password file <none>
I20260926 21:29:55.150111 930 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:55.150282 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:55.150652 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:55.153592 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "56ce6b7abd0448caa4fe9bfde5280998"
format_stamp: "Formatted at 2026-09-26 21:29:55 on dist-test-slave-87bt"
server_key: "78c7b3141d8b39950fdbce6da5374410"
server_key_iv: "bf92334ac38b7cceed61d28e33e30f71"
server_key_version: "encryptionkey@0"
I20260926 21:29:55.154374 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "56ce6b7abd0448caa4fe9bfde5280998"
format_stamp: "Formatted at 2026-09-26 21:29:55 on dist-test-slave-87bt"
server_key: "78c7b3141d8b39950fdbce6da5374410"
server_key_iv: "bf92334ac38b7cceed61d28e33e30f71"
server_key_version: "encryptionkey@0"
I20260926 21:29:55.159531 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:29:55.162819 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:55.164125 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:29:55.164340 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "56ce6b7abd0448caa4fe9bfde5280998"
format_stamp: "Formatted at 2026-09-26 21:29:55 on dist-test-slave-87bt"
server_key: "78c7b3141d8b39950fdbce6da5374410"
server_key_iv: "bf92334ac38b7cceed61d28e33e30f71"
server_key_version: "encryptionkey@0"
I20260926 21:29:55.164542 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:55.185796 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:55.189831 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:55.190131 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:55.208673 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37457
I20260926 21:29:55.208703 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37457 every 8 connection(s)
I20260926 21:29:55.210119 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:29:55.211562 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 930
I20260926 21:29:55.211795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:29:55.212431 388 external_mini_cluster.cc:1428] Setting key 52ed993e37a113bf25f1e4478f1d6e3a
I20260926 21:29:55.214963 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:29:55.227820 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998: Bootstrap starting.
Sep 26 21:29:55 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:29:55.231748 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998: Neither blocks nor log segments found. Creating new log.
I20260926 21:29:55.233070 997 log.cc:824] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998: Log is configured to *not* fsync() on all Append() calls
I20260926 21:29:55.236408 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998: No bootstrap required, opened a new log
I20260926 21:29:55.237380 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:55.214500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.97.62:37457)
0926 21:29:55.215297 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:55.215312 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:55.215362 (+ 50us) server_negotiation.cc:407] Connection header received
0926 21:29:55.217624 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:55.217650 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:55.218160 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:55.218628 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:55.219828 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:55.221827 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:55.223283 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:55.224322 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:55.224509 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:55.228908 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:55.228918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:55.228938 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:55.228980 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:55.232813 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:55.233540 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:55.233556 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:55.233568 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:55.233835 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:55.234243 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:55.234246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:55.234248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:55.234940 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:55.235207 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:55.235632 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 21:29:55.236027 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":318,"threads_started":1}
I20260926 21:29:55.244750 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ce6b7abd0448caa4fe9bfde5280998" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37457 } }
I20260926 21:29:55.245086 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:29:55.245157 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ce6b7abd0448caa4fe9bfde5280998, State: Initialized, Role: FOLLOWER
I20260926 21:29:55.245621 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ce6b7abd0448caa4fe9bfde5280998" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37457 } }
I20260926 21:29:55.245803 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:29:55.245905 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:29:55.246022 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:29:55.248138 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ce6b7abd0448caa4fe9bfde5280998" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37457 } }
I20260926 21:29:55.248558 997 leader_election.cc:302] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56ce6b7abd0448caa4fe9bfde5280998; no voters:
I20260926 21:29:55.249065 997 leader_election.cc:288] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:29:55.249213 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:29:55.249891 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [term 1 LEADER]: Becoming Leader. State: Replica: 56ce6b7abd0448caa4fe9bfde5280998, State: Running, Role: LEADER
I20260926 21:29:55.250397 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ce6b7abd0448caa4fe9bfde5280998" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37457 } }
I20260926 21:29:55.251224 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:29:55.253294 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56ce6b7abd0448caa4fe9bfde5280998" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ce6b7abd0448caa4fe9bfde5280998" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37457 } } }
I20260926 21:29:55.254177 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [sys.catalog]: This master's current role is: LEADER
I20260926 21:29:55.254006 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56ce6b7abd0448caa4fe9bfde5280998. Latest consensus state: current_term: 1 leader_uuid: "56ce6b7abd0448caa4fe9bfde5280998" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ce6b7abd0448caa4fe9bfde5280998" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37457 } } }
I20260926 21:29:55.254279 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998 [sys.catalog]: This master's current role is: LEADER
I20260926 21:29:55.255379 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:29:55.260695 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:29:55.269816 1009 catalog_manager.cc:1347] Generated new cluster ID: 032bea3e93774152a852d87a182ba733
I20260926 21:29:55.269953 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:29:55.283931 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:29:55.285786 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:29:55.295006 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56ce6b7abd0448caa4fe9bfde5280998: Generated new TSK 0
I20260926 21:29:55.296099 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:29:55.341825 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37457
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:55.665130 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:55.665398 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:55.665501 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:55.673151 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:55.673269 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:55.673380 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:29:55.683028 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:55.684222 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:55.685474 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:55.698495 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:55.698565 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:55.698587 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:55.699322 1023 server_base.cc:1034] running on GCE node
I20260926 21:29:55.699883 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:55.700770 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:55.701965 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790458195701921 us; error 35 us; skew 500 ppm
I20260926 21:29:55.702234 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:55 dist-test-slave-87bt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:55.705283 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:29:55.706750 1023 webserver.cc:460] Webserver started at http://127.0.97.1:33777/ using document root <none> and password file <none>
I20260926 21:29:55.707423 1023 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:55.707548 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:55.707854 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:55.710843 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "f26609e4a22a498d933ada0b121b0615"
format_stamp: "Formatted at 2026-09-26 21:29:55 on dist-test-slave-87bt"
server_key: "63215bb72db97270781bff94da0a69ca"
server_key_iv: "a0ec1b8735c64c4ca2730f5275da67e5"
server_key_version: "encryptionkey@0"
I20260926 21:29:55.711880 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f26609e4a22a498d933ada0b121b0615"
format_stamp: "Formatted at 2026-09-26 21:29:55 on dist-test-slave-87bt"
server_key: "63215bb72db97270781bff94da0a69ca"
server_key_iv: "a0ec1b8735c64c4ca2730f5275da67e5"
server_key_version: "encryptionkey@0"
I20260926 21:29:55.716758 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:29:55.720055 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:55.721251 1023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 21:29:55.721423 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "f26609e4a22a498d933ada0b121b0615"
format_stamp: "Formatted at 2026-09-26 21:29:55 on dist-test-slave-87bt"
server_key: "63215bb72db97270781bff94da0a69ca"
server_key_iv: "a0ec1b8735c64c4ca2730f5275da67e5"
server_key_version: "encryptionkey@0"
I20260926 21:29:55.721616 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:55.746372 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:55.750250 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:55.750592 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:55.751590 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:29:55.752949 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:29:55.753048 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:55.753190 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:29:55.753259 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:55.786288 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43937
I20260926 21:29:55.786335 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43937 every 8 connection(s)
I20260926 21:29:55.787590 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:29:55.788283 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 1023
I20260926 21:29:55.788533 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:29:55.789239 388 external_mini_cluster.cc:1428] Setting key 490b719d0793585a5231d5bef02043e0
Sep 26 21:29:55 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:29:55.809300 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:55.790359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53531 (local address 127.0.97.62:37457)
0926 21:29:55.790795 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:55.790801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:55.792112 (+ 1311us) server_negotiation.cc:407] Connection header received
0926 21:29:55.795072 (+ 2960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:55.795078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:55.795182 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:55.795315 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:55.797077 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:55.797792 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:55.799368 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:55.799779 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:55.799911 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:55.803402 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:55.803408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:55.803424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:55.803456 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:55.806026 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:55.806858 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:55.806864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:55.806867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:55.806990 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:55.807464 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:55.807469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:55.807473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:55.807786 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:55.807945 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:55.808861 (+ 916us) server_negotiation.cc:299] Negotiation successful
0926 21:29:55.809049 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":220,"threads_started":1}
I20260926 21:29:55.811026 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:55.791069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37457 (local address 127.0.97.1:53531)
0926 21:29:55.791908 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:55.791992 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:55.794756 (+ 2764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:55.795569 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:55.795582 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:55.796333 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:29:55.796893 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:55.796926 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:55.797955 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:55.797959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:55.799228 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:55.799239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:55.800008 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:55.800016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:55.800233 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:55.800909 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:29:55.800937 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:29:55.803181 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:29:55.806174 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:55.806184 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:55.806203 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:55.806726 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:55.807114 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:55.807119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:55.807123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:55.807333 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:55.807950 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:29:55.807960 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:29:55.808416 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 21:29:55.808738 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 21:29:55.809368 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":316,"threads_started":1}
I20260926 21:29:55.812603 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37457
I20260926 21:29:55.812979 1151 heartbeater.cc:464] Registering TS with master...
I20260926 21:29:55.813849 1151 heartbeater.cc:511] Master 127.0.97.62:37457 requested a full tablet report, sending...
I20260926 21:29:55.815963 962 ts_manager.cc:194] Registered new tserver with Master: f26609e4a22a498d933ada0b121b0615 (127.0.97.1:43937)
I20260926 21:29:55.817327 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:29:55.829466 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37457
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:56.141798 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:56.142082 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:56.142241 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:56.149631 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:56.149775 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:56.149966 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:29:56.160197 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:56.161504 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:56.162851 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:56.176349 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:56.176272 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:56.176308 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:56.177109 1157 server_base.cc:1034] running on GCE node
I20260926 21:29:56.177783 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:56.178861 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:56.180091 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790458196180038 us; error 43 us; skew 500 ppm
I20260926 21:29:56.180418 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:56 dist-test-slave-87bt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:56.184149 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:29:56.185674 1157 webserver.cc:460] Webserver started at http://127.0.97.2:33243/ using document root <none> and password file <none>
I20260926 21:29:56.186396 1157 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:56.186606 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:56.186940 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:56.189877 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "d811ee58b5fd4743aa7268fc95a6d9b9"
format_stamp: "Formatted at 2026-09-26 21:29:56 on dist-test-slave-87bt"
server_key: "0c5537f89a265aa02af34ff0e18a51fc"
server_key_iv: "fde323853879bef26e6b489d2101be54"
server_key_version: "encryptionkey@0"
I20260926 21:29:56.190747 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d811ee58b5fd4743aa7268fc95a6d9b9"
format_stamp: "Formatted at 2026-09-26 21:29:56 on dist-test-slave-87bt"
server_key: "0c5537f89a265aa02af34ff0e18a51fc"
server_key_iv: "fde323853879bef26e6b489d2101be54"
server_key_version: "encryptionkey@0"
I20260926 21:29:56.196121 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:29:56.199607 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:56.201036 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:29:56.201221 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "d811ee58b5fd4743aa7268fc95a6d9b9"
format_stamp: "Formatted at 2026-09-26 21:29:56 on dist-test-slave-87bt"
server_key: "0c5537f89a265aa02af34ff0e18a51fc"
server_key_iv: "fde323853879bef26e6b489d2101be54"
server_key_version: "encryptionkey@0"
I20260926 21:29:56.201427 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:56.216751 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:56.221063 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:56.221360 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:56.222273 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:29:56.223558 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:29:56.223664 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:56.223825 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:29:56.223897 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:56.255802 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40455
I20260926 21:29:56.255856 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40455 every 8 connection(s)
I20260926 21:29:56.257182 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:29:56.265465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 1157
I20260926 21:29:56.265658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:29:56.266152 388 external_mini_cluster.cc:1428] Setting key 267f1dd2b00c708a00d965dacba07bd6
Sep 26 21:29:56 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:29:56.278478 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.259596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41513 (local address 127.0.97.62:37457)
0926 21:29:56.259760 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:56.259764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:56.261032 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 21:29:56.262853 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:56.262857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:56.262928 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:56.263017 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:56.264394 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.265093 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.267179 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.267641 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.267781 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.271754 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:56.271759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:56.271763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:56.271793 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:56.275317 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:56.275992 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:56.276000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:56.276007 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:56.276151 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:56.276555 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:56.276562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:56.276569 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:56.276934 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:56.277080 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:56.277705 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 21:29:56.277959 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260926 21:29:56.279134 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.260100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37457 (local address 127.0.97.2:41513)
0926 21:29:56.260863 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:56.260911 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:56.262612 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:56.263213 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:56.263220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:56.263707 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:29:56.264258 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.264274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.265239 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.265246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.267053 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.267065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.267899 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.267907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.268129 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.268831 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:29:56.268853 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:29:56.271521 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:29:56.275446 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:56.275452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:56.275465 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:56.275859 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:56.276262 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:56.276265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:56.276267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:56.276429 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:56.277050 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:29:56.277055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:29:56.277357 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 21:29:56.277566 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 21:29:56.277935 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":286,"threads_started":1}
I20260926 21:29:56.281579 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37457
I20260926 21:29:56.281944 1285 heartbeater.cc:464] Registering TS with master...
I20260926 21:29:56.283010 1285 heartbeater.cc:511] Master 127.0.97.62:37457 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:29:56.284711 962 ts_manager.cc:194] Registered new tserver with Master: d811ee58b5fd4743aa7268fc95a6d9b9 (127.0.97.2:40455)
I20260926 21:29:56.285631 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:29:56.306951 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37457
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:56.622481 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:56.622804 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:56.622959 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:56.630185 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:56.630334 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:56.630532 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:29:56.640093 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:56.641377 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:56.642721 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:56.655836 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:56.655885 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:56.655977 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:56.656939 1290 server_base.cc:1034] running on GCE node
I20260926 21:29:56.657541 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:56.658497 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:56.659694 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790458196659655 us; error 40 us; skew 500 ppm
I20260926 21:29:56.659996 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:56 dist-test-slave-87bt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:56.663280 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:29:56.665452 1290 webserver.cc:460] Webserver started at http://127.0.97.3:41501/ using document root <none> and password file <none>
I20260926 21:29:56.666316 1290 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:56.666487 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:56.666890 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:56.669909 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "04055058ddc54bf5b8f149f08ce0d436"
format_stamp: "Formatted at 2026-09-26 21:29:56 on dist-test-slave-87bt"
server_key: "e99da6bc2a04c0e866445ce50d094988"
server_key_iv: "c8bbc81a8dd577f518a6c27e70d58737"
server_key_version: "encryptionkey@0"
I20260926 21:29:56.670706 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "04055058ddc54bf5b8f149f08ce0d436"
format_stamp: "Formatted at 2026-09-26 21:29:56 on dist-test-slave-87bt"
server_key: "e99da6bc2a04c0e866445ce50d094988"
server_key_iv: "c8bbc81a8dd577f518a6c27e70d58737"
server_key_version: "encryptionkey@0"
I20260926 21:29:56.675614 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:29:56.678933 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:56.680217 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:29:56.680393 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "04055058ddc54bf5b8f149f08ce0d436"
format_stamp: "Formatted at 2026-09-26 21:29:56 on dist-test-slave-87bt"
server_key: "e99da6bc2a04c0e866445ce50d094988"
server_key_iv: "c8bbc81a8dd577f518a6c27e70d58737"
server_key_version: "encryptionkey@0"
I20260926 21:29:56.680644 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:56.692621 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:56.697075 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:56.697383 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:56.698312 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:29:56.699627 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:29:56.699699 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:56.699823 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:29:56.699874 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:56.731962 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46083
I20260926 21:29:56.732050 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46083 every 8 connection(s)
I20260926 21:29:56.733191 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:29:56.741850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 1290
I20260926 21:29:56.742062 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:29:56.742528 388 external_mini_cluster.cc:1428] Setting key c3b78c96002eeac24c6e76cf272363a2
Sep 26 21:29:56 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:29:56.752276 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.735771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48727 (local address 127.0.97.62:37457)
0926 21:29:56.735944 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:56.735948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:56.737029 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 21:29:56.738890 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:56.738896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:56.738987 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:56.739096 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:56.740537 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.741262 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.743007 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.743602 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.743750 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.747209 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:56.747215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:56.747218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:56.747252 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:56.749570 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:56.750220 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:56.750224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:56.750226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:56.750339 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:56.750745 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:56.750749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:56.750751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:56.751012 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:56.751124 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:56.751860 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 21:29:56.752033 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:29:56.753219 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.736212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37457 (local address 127.0.97.3:48727)
0926 21:29:56.736873 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:56.736913 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:56.738627 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:56.739336 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:56.739344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:56.739827 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:29:56.740372 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.740389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.741422 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.741426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.742854 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.742870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.743850 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.743857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.744053 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.744770 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:29:56.744791 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:29:56.747008 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:29:56.749697 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:56.749704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:56.749718 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:56.750117 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:56.750473 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:56.750477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:56.750479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:56.750644 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:56.751118 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:29:56.751124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:29:56.751505 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 21:29:56.751743 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 21:29:56.752130 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260926 21:29:56.754649 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37457
I20260926 21:29:56.754972 1418 heartbeater.cc:464] Registering TS with master...
I20260926 21:29:56.755679 1418 heartbeater.cc:511] Master 127.0.97.62:37457 requested a full tablet report, sending...
I20260926 21:29:56.757477 962 ts_manager.cc:194] Registered new tserver with Master: 04055058ddc54bf5b8f149f08ce0d436 (127.0.97.3:46083)
I20260926 21:29:56.758574 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48727
I20260926 21:29:56.769821 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:29:56.789084 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.776306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.97.62:37457)
0926 21:29:56.776471 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:56.776475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:56.776899 (+ 424us) server_negotiation.cc:407] Connection header received
0926 21:29:56.777038 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:56.777042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:56.777100 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:56.777188 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:56.778646 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.779354 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.780542 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.780953 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.781050 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.783363 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:56.783368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:56.783371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:56.783405 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:56.786292 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:56.787031 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:56.787040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:56.787046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:56.787201 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:56.787583 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:56.787587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:56.787590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:56.788005 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:56.788261 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:56.788539 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 21:29:56.788752 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:29:56.795926 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:29:56.802891 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:29:56.821034 1151 heartbeater.cc:503] Master 127.0.97.62:37457 was elected leader, sending a full tablet report...
I20260926 21:29:56.842039 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.817525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.97.2:40455)
0926 21:29:56.829120 (+ 11595us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:56.829132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:56.832296 (+ 3164us) server_negotiation.cc:407] Connection header received
0926 21:29:56.832664 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:56.832672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:56.832893 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:56.833071 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:29:56.833999 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.835620 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.839014 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.840504 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.840609 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.840817 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:56.841560 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 21:29:56.841723 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11420,"thread_start_us":11266,"threads_started":1}
I20260926 21:29:56.842069 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.814809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46083 (local address 127.0.0.1:42728)
0926 21:29:56.830475 (+ 15666us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:56.830527 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:56.831673 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:56.834376 (+ 2703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:56.834381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:56.834404 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:29:56.834729 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.834740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.836602 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.836607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.838823 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.838839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.841023 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.841032 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.841267 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.841326 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 21:29:56.841459 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 21:29:56.841654 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15360,"mutex_wait_us":2,"thread_start_us":15188,"threads_started":1}
I20260926 21:29:56.843088 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.814998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.0.97.3:46083)
0926 21:29:56.829823 (+ 14825us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:56.829849 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:56.830665 (+ 816us) server_negotiation.cc:407] Connection header received
0926 21:29:56.833672 (+ 3007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:56.833684 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:56.833939 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:56.834119 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:29:56.834922 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.836401 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.839019 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.840504 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.840622 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.840855 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:56.842579 (+ 1724us) server_negotiation.cc:299] Negotiation successful
0926 21:29:56.842749 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14497,"thread_start_us":14328,"threads_started":1}
I20260926 21:29:56.843358 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.817271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40455 (local address 127.0.0.1:38312)
0926 21:29:56.832164 (+ 14893us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:56.832180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:56.832437 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:56.833363 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:56.833368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:56.833393 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:29:56.833800 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.833812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.835803 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.835807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.838830 (+ 3023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.838839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.841188 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.841197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.841354 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.841377 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:29:56.842710 (+ 1333us) client_negotiation.cc:240] Negotiation successful
0926 21:29:56.842840 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14763,"mutex_wait_us":43,"thread_start_us":14633,"threads_started":1}
I20260926 21:29:56.844857 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.815072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43937 (local address 127.0.0.1:55090)
0926 21:29:56.831130 (+ 16058us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:56.831149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:56.831432 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:56.832852 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:56.832862 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:56.832901 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:29:56.833363 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.833378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.835552 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.835558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.841292 (+ 5734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:56.841306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.844025 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.844034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:56.844215 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.844249 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:29:56.844361 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 21:29:56.844495 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15912,"mutex_wait_us":1471,"thread_start_us":15773,"threads_started":1}
I20260926 21:29:56.844934 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:56.815331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.0.97.1:43937)
0926 21:29:56.828410 (+ 13079us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:56.828425 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:56.831249 (+ 2824us) server_negotiation.cc:407] Connection header received
0926 21:29:56.831909 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:56.831923 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:56.832227 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:56.832456 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:29:56.833653 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.835367 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.841475 (+ 6108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:56.843003 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:56.843105 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:56.843764 (+ 659us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:56.844433 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 21:29:56.844599 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12833,"thread_start_us":13099,"threads_started":1}
I20260926 21:29:56.847288 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 13b7aabab1de45ff9533a347c193c706 (DEFAULT_TABLE table=test-table [id=5188e0f1e3314c7093fe7dd6e129d84b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:29:56.847288 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 13b7aabab1de45ff9533a347c193c706 (DEFAULT_TABLE table=test-table [id=5188e0f1e3314c7093fe7dd6e129d84b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:29:56.848762 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b7aabab1de45ff9533a347c193c706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:29:56.848766 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b7aabab1de45ff9533a347c193c706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:29:56.848555 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 13b7aabab1de45ff9533a347c193c706 (DEFAULT_TABLE table=test-table [id=5188e0f1e3314c7093fe7dd6e129d84b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:29:56.849877 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b7aabab1de45ff9533a347c193c706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:29:56.867841 1437 tablet_bootstrap.cc:492] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615: Bootstrap starting.
I20260926 21:29:56.868383 1436 tablet_bootstrap.cc:492] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9: Bootstrap starting.
I20260926 21:29:56.869199 1438 tablet_bootstrap.cc:492] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436: Bootstrap starting.
I20260926 21:29:56.871224 1437 tablet_bootstrap.cc:654] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615: Neither blocks nor log segments found. Creating new log.
I20260926 21:29:56.872344 1437 log.cc:824] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615: Log is configured to *not* fsync() on all Append() calls
I20260926 21:29:56.873099 1438 tablet_bootstrap.cc:654] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436: Neither blocks nor log segments found. Creating new log.
I20260926 21:29:56.873103 1436 tablet_bootstrap.cc:654] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9: Neither blocks nor log segments found. Creating new log.
I20260926 21:29:56.874248 1438 log.cc:824] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436: Log is configured to *not* fsync() on all Append() calls
I20260926 21:29:56.874702 1436 log.cc:824] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9: Log is configured to *not* fsync() on all Append() calls
I20260926 21:29:56.875384 1437 tablet_bootstrap.cc:492] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615: No bootstrap required, opened a new log
I20260926 21:29:56.875685 1437 ts_tablet_manager.cc:1397] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 21:29:56.878058 1436 tablet_bootstrap.cc:492] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9: No bootstrap required, opened a new log
I20260926 21:29:56.878332 1436 ts_tablet_manager.cc:1397] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260926 21:29:56.878624 1438 tablet_bootstrap.cc:492] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436: No bootstrap required, opened a new log
I20260926 21:29:56.878892 1438 ts_tablet_manager.cc:1397] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 21:29:56.886139 1436 raft_consensus.cc:348] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:56.886564 1436 raft_consensus.cc:374] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:29:56.886704 1436 raft_consensus.cc:729] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d811ee58b5fd4743aa7268fc95a6d9b9, State: Initialized, Role: FOLLOWER
I20260926 21:29:56.886687 1437 raft_consensus.cc:348] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:56.887179 1437 raft_consensus.cc:374] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:29:56.887292 1437 raft_consensus.cc:729] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26609e4a22a498d933ada0b121b0615, State: Initialized, Role: FOLLOWER
I20260926 21:29:56.887358 1436 consensus_queue.cc:260] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:56.887852 1437 consensus_queue.cc:260] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:56.888566 1285 heartbeater.cc:503] Master 127.0.97.62:37457 was elected leader, sending a full tablet report...
I20260926 21:29:56.889237 1436 ts_tablet_manager.cc:1428] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 21:29:56.890161 1438 raft_consensus.cc:348] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:56.890627 1438 raft_consensus.cc:374] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:29:56.890744 1438 raft_consensus.cc:729] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04055058ddc54bf5b8f149f08ce0d436, State: Initialized, Role: FOLLOWER
I20260926 21:29:56.890987 1437 ts_tablet_manager.cc:1428] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 21:29:56.891392 1438 consensus_queue.cc:260] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:56.892742 1418 heartbeater.cc:503] Master 127.0.97.62:37457 was elected leader, sending a full tablet report...
I20260926 21:29:56.893314 1438 ts_tablet_manager.cc:1428] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260926 21:29:56.984694 1419 tablet.cc:2369] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:29:57.009075 1286 tablet.cc:2369] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:29:57.011293 1442 raft_consensus.cc:482] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:29:57.011575 1442 raft_consensus.cc:504] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:57.013705 1442 leader_election.cc:288] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04055058ddc54bf5b8f149f08ce0d436 (127.0.97.3:46083), f26609e4a22a498d933ada0b121b0615 (127.0.97.1:43937)
I20260926 21:29:57.020087 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.013899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43897 (local address 127.0.97.1:43937)
0926 21:29:57.014106 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:57.014114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:57.015108 (+ 994us) server_negotiation.cc:407] Connection header received
0926 21:29:57.015301 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:57.015307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:57.015372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:57.015468 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:29:57.015990 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.016672 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.018357 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.019185 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.019342 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.019447 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:57.019720 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 21:29:57.019849 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:29:57.020332 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.013771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58055 (local address 127.0.97.3:46083)
0926 21:29:57.013994 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:57.014004 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:57.014714 (+ 710us) server_negotiation.cc:407] Connection header received
0926 21:29:57.014961 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:57.014967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:57.015034 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:57.015130 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:29:57.015617 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.016362 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.018158 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.018979 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.019126 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.019232 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:57.019514 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 21:29:57.019635 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:29:57.020401 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.014228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43937 (local address 127.0.97.2:43897)
0926 21:29:57.014999 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:57.015013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:57.015172 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:57.015485 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:57.015489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:57.015510 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:29:57.015850 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:57.015858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.016811 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.016817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:57.018201 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:57.018213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.019403 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.019409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:57.019569 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.019591 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:29:57.019685 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 21:29:57.019807 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":1,"thread_start_us":466,"threads_started":1}
I20260926 21:29:57.020453 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.013829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46083 (local address 127.0.97.2:58055)
0926 21:29:57.014592 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:57.014607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:57.014810 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:57.015150 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:57.015155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:57.015181 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:29:57.015484 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:57.015492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.016493 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.016499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:57.018013 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:57.018025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.019190 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.019196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:57.019354 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.019377 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:29:57.019491 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 21:29:57.019621 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":24,"thread_start_us":385,"threads_started":1}
I20260926 21:29:57.021450 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b7aabab1de45ff9533a347c193c706" candidate_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26609e4a22a498d933ada0b121b0615" is_pre_election: true
I20260926 21:29:57.021513 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b7aabab1de45ff9533a347c193c706" candidate_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04055058ddc54bf5b8f149f08ce0d436" is_pre_election: true
I20260926 21:29:57.021957 1373 raft_consensus.cc:2454] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d811ee58b5fd4743aa7268fc95a6d9b9 in term 0.
I20260926 21:29:57.021955 1106 raft_consensus.cc:2454] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d811ee58b5fd4743aa7268fc95a6d9b9 in term 0.
I20260926 21:29:57.022634 1175 leader_election.cc:302] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d811ee58b5fd4743aa7268fc95a6d9b9, f26609e4a22a498d933ada0b121b0615; no voters:
I20260926 21:29:57.023028 1442 raft_consensus.cc:2781] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:29:57.023169 1442 raft_consensus.cc:482] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:29:57.023288 1442 raft_consensus.cc:3037] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:29:57.025414 1442 raft_consensus.cc:504] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:57.026098 1442 leader_election.cc:288] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [CANDIDATE]: Term 1 election: Requested vote from peers 04055058ddc54bf5b8f149f08ce0d436 (127.0.97.3:46083), f26609e4a22a498d933ada0b121b0615 (127.0.97.1:43937)
I20260926 21:29:57.026576 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b7aabab1de45ff9533a347c193c706" candidate_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04055058ddc54bf5b8f149f08ce0d436"
I20260926 21:29:57.026621 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b7aabab1de45ff9533a347c193c706" candidate_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26609e4a22a498d933ada0b121b0615"
I20260926 21:29:57.026801 1373 raft_consensus.cc:3037] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:29:57.026819 1106 raft_consensus.cc:3037] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:29:57.029960 1373 raft_consensus.cc:2454] T 13b7aabab1de45ff9533a347c193c706 P 04055058ddc54bf5b8f149f08ce0d436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d811ee58b5fd4743aa7268fc95a6d9b9 in term 1.
I20260926 21:29:57.029963 1106 raft_consensus.cc:2454] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d811ee58b5fd4743aa7268fc95a6d9b9 in term 1.
I20260926 21:29:57.030450 1174 leader_election.cc:302] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04055058ddc54bf5b8f149f08ce0d436, d811ee58b5fd4743aa7268fc95a6d9b9, f26609e4a22a498d933ada0b121b0615; no voters:
I20260926 21:29:57.030790 1442 raft_consensus.cc:2781] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:29:57.031224 1442 raft_consensus.cc:686] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [term 1 LEADER]: Becoming Leader. State: Replica: d811ee58b5fd4743aa7268fc95a6d9b9, State: Running, Role: LEADER
I20260926 21:29:57.031739 1442 consensus_queue.cc:237] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } }
I20260926 21:29:57.036237 960 catalog_manager.cc:5495] T 13b7aabab1de45ff9533a347c193c706 P d811ee58b5fd4743aa7268fc95a6d9b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d811ee58b5fd4743aa7268fc95a6d9b9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04055058ddc54bf5b8f149f08ce0d436" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26609e4a22a498d933ada0b121b0615" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d811ee58b5fd4743aa7268fc95a6d9b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } health_report { overall_health: HEALTHY } } }
W20260926 21:29:57.040058 1152 tablet.cc:2369] T 13b7aabab1de45ff9533a347c193c706 P f26609e4a22a498d933ada0b121b0615: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:29:57.097270 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.082677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.0.97.1:43937)
0926 21:29:57.082870 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:57.082877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:57.083318 (+ 441us) server_negotiation.cc:407] Connection header received
0926 21:29:57.083507 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:57.083514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:57.083624 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:57.083726 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:57.085291 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.086035 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.087336 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.087756 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.087900 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.091904 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:57.091918 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:57.091934 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:57.091972 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:57.095025 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.095665 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.095671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.095673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.095785 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.096089 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.096092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.096094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.096425 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:57.096631 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:57.096870 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 21:29:57.097022 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:29:57.141506 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.127782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.0.97.1:43937)
0926 21:29:57.127982 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:57.127992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:57.128320 (+ 328us) server_negotiation.cc:407] Connection header received
0926 21:29:57.128527 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:57.128537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:57.128601 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:57.128706 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:57.130040 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.130828 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.132112 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.132529 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.132663 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.136464 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:57.136469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:57.136472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:57.136506 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:57.139486 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.140035 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.140038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.140040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.140159 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.140483 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.140486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.140488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.140723 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:57.140904 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:57.141123 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 21:29:57.141276 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:29:57.185364 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.170632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.0.97.1:43937)
0926 21:29:57.170876 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:57.170884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:57.171109 (+ 225us) server_negotiation.cc:407] Connection header received
0926 21:29:57.171317 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:57.171323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:57.171447 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:57.171551 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:57.173003 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.173683 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.175102 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.175536 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.175669 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.179618 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:57.179623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:57.179626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:57.179659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:57.182229 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.182947 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.182957 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.182960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.183070 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.183447 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.183452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.183455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.183828 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:57.184082 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:57.184384 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 21:29:57.184616 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 21:29:57.187219 1066 tablet_service.cc:2908] Rejecting scan request for tablet 13b7aabab1de45ff9533a347c193c706: Uninitialized: clean time has not yet been initialized
I20260926 21:29:57.190018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1023
I20260926 21:29:57.202287 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.190996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.0.97.1:43937)
0926 21:29:57.191197 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:57.191204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:57.191612 (+ 408us) server_negotiation.cc:407] Connection header received
0926 21:29:57.191738 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:57.191742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:57.191813 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:57.191911 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:57.192976 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.193703 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.194933 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.195512 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.195628 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.197662 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:57.197667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:57.197671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:57.197700 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:57.200057 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.200622 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.200626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.200628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.200732 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.201028 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.201031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.201033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.201322 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:57.201520 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:57.201729 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 21:29:57.201915 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:29:57.203052 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:29:57.301308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 1023
I20260926 21:29:57.317984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1157
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:29:57.331509 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.318793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.0.97.2:40455)
0926 21:29:57.318967 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:57.318971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:57.319005 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:29:57.319143 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:57.319147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:57.319210 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:57.319305 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:57.320386 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.321092 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.322267 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.322824 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.322915 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.326679 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:57.326691 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:57.326707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:57.326739 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:57.329232 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.329804 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.329811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.329813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.329934 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.330230 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.330233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.330235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.330705 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:57.330935 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:57.331095 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 21:29:57.331282 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 21:29:57.332187 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:29:57.421952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 1157
I20260926 21:29:57.438225 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1290
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:29:57.452478 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:57.438971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.0.97.3:46083)
0926 21:29:57.439141 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:57.439149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:57.439170 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:29:57.439380 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:57.439386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:57.439465 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:57.439572 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:57.440737 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.441442 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.442732 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:57.443117 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:57.443251 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:57.447332 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:57.447346 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:57.447363 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:57.447396 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:57.449845 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.450471 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.450481 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.450485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.450622 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:57.450971 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:57.450976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:57.450978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:57.451479 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:57.451706 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:57.452019 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 21:29:57.452225 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:29:57.453063 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:29:57.538033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 1290
I20260926 21:29:57.554272 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 930
I20260926 21:29:57.555402 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:29:57.660094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 930
2026-09-26T21:29:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4112 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[1466](info): set up 2 sockets
Sep 26 21:29:57 dist-test-slave-87bt krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260926 21:29:58.750758 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:29:58 dist-test-slave-87bt krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:29:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:29:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:29:58.934069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34745
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:59.248049 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:59.248329 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:59.248502 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:59.255673 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:29:59.255775 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 21:29:59.255862 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:59.255986 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:59.256064 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:29:59.256150 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:29:59.266335 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:59.267630 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:59.269004 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:59.280835 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:59.280858 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:59.280771 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:59.281548 1479 server_base.cc:1034] running on GCE node
I20260926 21:29:59.282404 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:59.283946 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:59.285199 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790458199285136 us; error 34 us; skew 500 ppm
I20260926 21:29:59.285568 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:59 dist-test-slave-87bt krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:59.292619 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:29:59.294138 1479 webserver.cc:460] Webserver started at http://127.0.97.62:42661/ using document root <none> and password file <none>
I20260926 21:29:59.294920 1479 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:59.295053 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:59.295394 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:59.298877 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "2c765742b69a40e89838223f74da4f31"
format_stamp: "Formatted at 2026-09-26 21:29:59 on dist-test-slave-87bt"
server_key: "1aae665a7bae9d8c8bb504a09957ebeb"
server_key_iv: "e9f2c6a2c0563a058a3fc0ee44d564f6"
server_key_version: "encryptionkey@0"
I20260926 21:29:59.299664 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c765742b69a40e89838223f74da4f31"
format_stamp: "Formatted at 2026-09-26 21:29:59 on dist-test-slave-87bt"
server_key: "1aae665a7bae9d8c8bb504a09957ebeb"
server_key_iv: "e9f2c6a2c0563a058a3fc0ee44d564f6"
server_key_version: "encryptionkey@0"
I20260926 21:29:59.304728 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:29:59.308187 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:59.309475 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:29:59.309654 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "2c765742b69a40e89838223f74da4f31"
format_stamp: "Formatted at 2026-09-26 21:29:59 on dist-test-slave-87bt"
server_key: "1aae665a7bae9d8c8bb504a09957ebeb"
server_key_iv: "e9f2c6a2c0563a058a3fc0ee44d564f6"
server_key_version: "encryptionkey@0"
I20260926 21:29:59.309865 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:59.331903 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:59.336421 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:59.336702 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:59.358158 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34745
I20260926 21:29:59.358191 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34745 every 8 connection(s)
I20260926 21:29:59.359526 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:29:59.360185 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 1479
I20260926 21:29:59.360412 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:29:59.361212 388 external_mini_cluster.cc:1428] Setting key 30844c705184b7a6a19f2e8ab37dc1c1
I20260926 21:29:59.364156 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:29:59.374588 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31: Bootstrap starting.
Sep 26 21:29:59 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:29:59.377872 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31: Neither blocks nor log segments found. Creating new log.
I20260926 21:29:59.379133 1556 log.cc:824] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31: Log is configured to *not* fsync() on all Append() calls
I20260926 21:29:59.381925 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31: No bootstrap required, opened a new log
I20260926 21:29:59.384080 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:59.363156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.0.97.62:34745)
0926 21:29:59.364122 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:59.364136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:59.364186 (+ 50us) server_negotiation.cc:407] Connection header received
0926 21:29:59.365969 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:59.365992 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:59.366415 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:59.366972 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:59.368536 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.369922 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.371283 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.372071 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.372345 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:59.376708 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:59.376718 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:59.376734 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:59.376775 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:59.380097 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:59.380822 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:59.380829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:59.380836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:59.381005 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:59.381403 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:59.381407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:59.381409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:59.381946 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:59.382242 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:59.382697 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 21:29:59.383099 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":522,"threads_started":1}
I20260926 21:29:59.390359 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c765742b69a40e89838223f74da4f31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34745 } }
I20260926 21:29:59.390784 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:29:59.390897 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c765742b69a40e89838223f74da4f31, State: Initialized, Role: FOLLOWER
I20260926 21:29:59.391486 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c765742b69a40e89838223f74da4f31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34745 } }
I20260926 21:29:59.391769 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:29:59.391942 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:29:59.392117 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:29:59.394667 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c765742b69a40e89838223f74da4f31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34745 } }
I20260926 21:29:59.395146 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c765742b69a40e89838223f74da4f31; no voters:
I20260926 21:29:59.395673 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:29:59.395798 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:29:59.396477 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [term 1 LEADER]: Becoming Leader. State: Replica: 2c765742b69a40e89838223f74da4f31, State: Running, Role: LEADER
I20260926 21:29:59.396966 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c765742b69a40e89838223f74da4f31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34745 } }
I20260926 21:29:59.397650 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:29:59.400148 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c765742b69a40e89838223f74da4f31. Latest consensus state: current_term: 1 leader_uuid: "2c765742b69a40e89838223f74da4f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c765742b69a40e89838223f74da4f31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34745 } } }
I20260926 21:29:59.400597 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [sys.catalog]: This master's current role is: LEADER
I20260926 21:29:59.400272 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c765742b69a40e89838223f74da4f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c765742b69a40e89838223f74da4f31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34745 } } }
I20260926 21:29:59.401062 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31 [sys.catalog]: This master's current role is: LEADER
I20260926 21:29:59.402568 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:29:59.407516 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:29:59.415920 1570 catalog_manager.cc:1347] Generated new cluster ID: 029fa55ad2484416970a9e116907876c
I20260926 21:29:59.416038 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:29:59.422991 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:29:59.424352 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:29:59.433346 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31: Generated new TSK 0
I20260926 21:29:59.434314 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:29:59.486099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34745
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:29:59.810247 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:29:59.810603 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:29:59.810804 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:29:59.818207 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:29:59.818310 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:29:59.818508 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:29:59.818625 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:29:59.828365 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:29:59.829664 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:29:59.831058 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:29:59.843792 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:59.843757 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:29:59.843825 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:29:59.844695 1582 server_base.cc:1034] running on GCE node
I20260926 21:29:59.845218 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:29:59.846164 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:29:59.847374 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790458199847278 us; error 67 us; skew 500 ppm
I20260926 21:29:59.847685 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:29:59 dist-test-slave-87bt krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:29:59.850790 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:29:59.852289 1582 webserver.cc:460] Webserver started at http://127.0.97.1:37203/ using document root <none> and password file <none>
I20260926 21:29:59.852957 1582 fs_manager.cc:359] Metadata directory not provided
I20260926 21:29:59.853039 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:29:59.853364 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:29:59.856271 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "5619edcf42754ec3b8594c104ce37d6f"
format_stamp: "Formatted at 2026-09-26 21:29:59 on dist-test-slave-87bt"
server_key: "d748e0fd80dfbbf5252e08a4490879be"
server_key_iv: "c36594df23f65343d5cf8c1cbb055908"
server_key_version: "encryptionkey@0"
I20260926 21:29:59.857048 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5619edcf42754ec3b8594c104ce37d6f"
format_stamp: "Formatted at 2026-09-26 21:29:59 on dist-test-slave-87bt"
server_key: "d748e0fd80dfbbf5252e08a4490879be"
server_key_iv: "c36594df23f65343d5cf8c1cbb055908"
server_key_version: "encryptionkey@0"
I20260926 21:29:59.862015 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:29:59.865268 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:59.866623 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:29:59.866796 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "5619edcf42754ec3b8594c104ce37d6f"
format_stamp: "Formatted at 2026-09-26 21:29:59 on dist-test-slave-87bt"
server_key: "d748e0fd80dfbbf5252e08a4490879be"
server_key_iv: "c36594df23f65343d5cf8c1cbb055908"
server_key_version: "encryptionkey@0"
I20260926 21:29:59.867009 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:29:59.886361 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:29:59.891040 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:29:59.891320 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:29:59.893895 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:29:59.894004 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:29:59.894172 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:29:59.894253 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:29:59 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:29:59.918651 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:59.894636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59079 (local address 127.0.97.62:34745)
0926 21:29:59.895156 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:59.895162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:59.896333 (+ 1171us) server_negotiation.cc:407] Connection header received
0926 21:29:59.898556 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:59.898565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:59.898673 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:59.898805 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:59.901066 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.902402 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.905927 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.906724 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.906875 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:59.912085 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:59.912092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:59.912096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:59.912137 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:59.915240 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:59.916217 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:59.916222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:59.916226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:59.916379 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:59.916950 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:59.916955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:59.916961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:59.917222 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:59.917372 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:59.918036 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 21:29:59.918248 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":286,"threads_started":1}
I20260926 21:29:59.919750 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:59.895286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34745 (local address 127.0.97.1:59079)
0926 21:29:59.896156 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:59.896199 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:59.898139 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:59.899176 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:59.899192 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:59.899974 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:29:59.900834 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:59.900874 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.902773 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.902783 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:59.905676 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:59.905700 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.907260 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.907269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:59.907584 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:59.908373 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:29:59.908400 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:29:59.911840 (+ 3440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:29:59.915376 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:59.915384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:59.915404 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:59.916093 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:59.916529 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:59.916541 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:59.916545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:59.916826 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:59.917354 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:29:59.917361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:29:59.917704 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 21:29:59.917979 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 21:29:59.918514 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":386,"threads_started":1}
I20260926 21:29:59.944626 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:59.929831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37025 (local address 127.0.97.62:34745)
0926 21:29:59.930049 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:29:59.930058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:29:59.930688 (+ 630us) server_negotiation.cc:407] Connection header received
0926 21:29:59.930932 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:29:59.930938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:29:59.931028 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:29:59.931125 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:29:59.932520 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.933321 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.934671 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.935245 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.935376 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:59.938044 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:29:59.938048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:29:59.938051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:29:59.938079 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:29:59.941386 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:59.942464 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:59.942470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:59.942475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:59.942638 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:29:59.943080 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:29:59.943087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:29:59.943091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:29:59.943408 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:29:59.943580 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 21:29:59.943994 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 21:29:59.944216 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:29:59.944736 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:29:59.929805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34745 (local address 127.0.97.1:37025)
0926 21:29:59.930571 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 21:29:59.930595 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:29:59.930748 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:29:59.931267 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:29:59.931273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:29:59.931949 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:29:59.932388 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:59.932398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.933447 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.933454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:59.934540 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:29:59.934551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:29:59.935431 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:29:59.935439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:29:59.935620 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:29:59.936394 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:29:59.936426 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:29:59.937873 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:29:59.941541 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:59.941553 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:59.941559 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:59.942287 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:59.942796 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:29:59.942807 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:29:59.942811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:29:59.942959 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:29:59.943537 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:29:59.943542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:29:59.943732 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 21:29:59.943901 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 21:29:59.944226 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":384,"threads_started":1}
I20260926 21:29:59.950699 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35077
I20260926 21:29:59.950739 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35077 every 8 connection(s)
I20260926 21:29:59.951907 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:29:59.953743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 1582
I20260926 21:29:59.953960 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:29:59.953985 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34745
I20260926 21:29:59.954201 1718 heartbeater.cc:464] Registering TS with master...
I20260926 21:29:59.954792 388 external_mini_cluster.cc:1428] Setting key fd62cad7aaf591df0f04228e63225394
I20260926 21:29:59.955044 1718 heartbeater.cc:511] Master 127.0.97.62:34745 requested a full tablet report, sending...
I20260926 21:29:59.957403 1510 ts_manager.cc:194] Registered new tserver with Master: 5619edcf42754ec3b8594c104ce37d6f (127.0.97.1:35077)
I20260926 21:29:59.958920 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59079
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:29:59.993667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34745
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:00.311735 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:00.312033 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:00.312181 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:00.319542 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:00.319660 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:00.319829 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:30:00.319950 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:00.329813 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:00.331166 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:00.332564 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:00.345348 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:00.345319 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:00.345685 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:00.346122 1722 server_base.cc:1034] running on GCE node
I20260926 21:30:00.346709 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:00.347520 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:00.348775 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790458200348712 us; error 39 us; skew 500 ppm
I20260926 21:30:00.349097 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:00 dist-test-slave-87bt krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:00.352181 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:30:00.353674 1722 webserver.cc:460] Webserver started at http://127.0.97.2:43381/ using document root <none> and password file <none>
I20260926 21:30:00.354405 1722 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:00.354578 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:00.354894 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:00.357756 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "aa49273c753941d29eee0d2e55ca040c"
format_stamp: "Formatted at 2026-09-26 21:30:00 on dist-test-slave-87bt"
server_key: "21eac255c9a5ee277b1f7dccfa0ed8e2"
server_key_iv: "e3a22334e29d5e749ccce5df90782135"
server_key_version: "encryptionkey@0"
I20260926 21:30:00.358614 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aa49273c753941d29eee0d2e55ca040c"
format_stamp: "Formatted at 2026-09-26 21:30:00 on dist-test-slave-87bt"
server_key: "21eac255c9a5ee277b1f7dccfa0ed8e2"
server_key_iv: "e3a22334e29d5e749ccce5df90782135"
server_key_version: "encryptionkey@0"
I20260926 21:30:00.364032 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:30:00.367615 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:00.369140 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:30:00.369333 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "aa49273c753941d29eee0d2e55ca040c"
format_stamp: "Formatted at 2026-09-26 21:30:00 on dist-test-slave-87bt"
server_key: "21eac255c9a5ee277b1f7dccfa0ed8e2"
server_key_iv: "e3a22334e29d5e749ccce5df90782135"
server_key_version: "encryptionkey@0"
I20260926 21:30:00.369573 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:00.390976 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:00.395220 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:00.395539 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:00.398530 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:00.398670 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:00.398929 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:00.399029 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:30:00 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:00.424914 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.398955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54017 (local address 127.0.97.62:34745)
0926 21:30:00.399146 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:00.399155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:00.400557 (+ 1402us) server_negotiation.cc:407] Connection header received
0926 21:30:00.402748 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:00.402758 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:00.402871 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:00.403009 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:00.404976 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.406498 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.409567 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.410532 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.410688 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:00.416076 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:00.416084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:00.416092 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:00.416139 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:00.419970 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:00.421329 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:00.421336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:00.421340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:00.421532 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:00.422293 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:00.422300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:00.422303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:00.422754 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:00.422951 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:00.424106 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0926 21:30:00.424428 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 21:30:00.426573 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.399508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34745 (local address 127.0.97.2:54017)
0926 21:30:00.400374 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:00.400426 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:00.402402 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:00.403246 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:00.403254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:00.403778 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:00.404672 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:00.404695 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.406782 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.406796 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:00.409405 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:00.409419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.410816 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.410822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:00.411031 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:00.411777 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:00.411819 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:00.415739 (+ 3920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:00.420182 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:00.420199 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:00.420225 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:00.421164 (+ 939us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:00.421694 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:00.421703 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:00.421709 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:00.422136 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:00.423053 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:00.423060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:00.423554 (+ 494us) client_negotiation.cc:769] Sending connection context
0926 21:30:00.423954 (+ 400us) client_negotiation.cc:240] Negotiation successful
0926 21:30:00.424650 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":352,"threads_started":1}
I20260926 21:30:00.442588 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35063
I20260926 21:30:00.442687 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35063 every 8 connection(s)
I20260926 21:30:00.444286 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:00.446542 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34745
I20260926 21:30:00.446785 1857 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:00.447568 1857 heartbeater.cc:511] Master 127.0.97.62:34745 requested a full tablet report, sending...
I20260926 21:30:00.448956 1510 ts_manager.cc:194] Registered new tserver with Master: aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063)
I20260926 21:30:00.449862 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54017
I20260926 21:30:00.450397 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.434017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42219 (local address 127.0.97.62:34745)
0926 21:30:00.434214 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:00.434221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:00.434976 (+ 755us) server_negotiation.cc:407] Connection header received
0926 21:30:00.435174 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:00.435179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:00.435265 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:00.435406 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:00.437292 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.438658 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.440916 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.441595 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.441754 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:00.444299 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:00.444303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:00.444306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:00.444335 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:00.447715 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:00.448350 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:00.448357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:00.448360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:00.448489 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:00.448847 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:00.448853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:00.448857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:00.449261 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:00.449449 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:00.449822 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 21:30:00.450041 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:30:00.450526 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.434096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34745 (local address 127.0.97.2:42219)
0926 21:30:00.434862 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:00.434888 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:00.435033 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:00.435599 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:00.435607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:00.436511 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:00.437095 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:00.437113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.438875 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.438882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:00.440781 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:00.440792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.442038 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.442048 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:00.442248 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:00.443065 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:00.443086 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:00.444115 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:00.447832 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:00.447835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:00.447838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:00.448223 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:00.448601 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:00.448607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:00.448610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:00.448737 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:00.449419 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:00.449424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:00.449596 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 21:30:00.449739 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 21:30:00.450041 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":483,"threads_started":1}
I20260926 21:30:00.451061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 1722
I20260926 21:30:00.451253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:00.451701 388 external_mini_cluster.cc:1428] Setting key 0bc0e87fe38fc40d513557e6d024f2c8
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:30:00.487716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34745
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:00.805688 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:00.805982 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:00.806084 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:00.813356 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:00.813473 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:00.813556 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:30:00.813638 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:00.823735 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:00.825013 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:00.826340 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:00.842268 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:00.842394 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:00.842288 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:00.848120 1861 server_base.cc:1034] running on GCE node
I20260926 21:30:00.848802 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:00.849795 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:00.851040 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790458200850952 us; error 62 us; skew 500 ppm
I20260926 21:30:00.851356 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:00 dist-test-slave-87bt krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:00.854651 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:30:00.856168 1861 webserver.cc:460] Webserver started at http://127.0.97.3:46583/ using document root <none> and password file <none>
I20260926 21:30:00.856880 1861 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:00.857014 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:00.857334 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:00.860273 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "c770813c057347c8a6cacd6878213cb7"
format_stamp: "Formatted at 2026-09-26 21:30:00 on dist-test-slave-87bt"
server_key: "2ea87b73bbbb9539518d2ad9d0c2f967"
server_key_iv: "eb9a4ea9dae4f8731b90acfb12b85cdd"
server_key_version: "encryptionkey@0"
I20260926 21:30:00.861075 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c770813c057347c8a6cacd6878213cb7"
format_stamp: "Formatted at 2026-09-26 21:30:00 on dist-test-slave-87bt"
server_key: "2ea87b73bbbb9539518d2ad9d0c2f967"
server_key_iv: "eb9a4ea9dae4f8731b90acfb12b85cdd"
server_key_version: "encryptionkey@0"
I20260926 21:30:00.866022 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:30:00.869441 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:00.870837 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:30:00.871026 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "c770813c057347c8a6cacd6878213cb7"
format_stamp: "Formatted at 2026-09-26 21:30:00 on dist-test-slave-87bt"
server_key: "2ea87b73bbbb9539518d2ad9d0c2f967"
server_key_iv: "eb9a4ea9dae4f8731b90acfb12b85cdd"
server_key_version: "encryptionkey@0"
I20260926 21:30:00.871260 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:00.888559 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:00.892827 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:00.893122 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:00.895896 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:00.896025 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:00.896226 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:00.896356 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 21:30:00 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:00.922510 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.896347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33207 (local address 127.0.97.62:34745)
0926 21:30:00.896542 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:00.896548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:00.898224 (+ 1676us) server_negotiation.cc:407] Connection header received
0926 21:30:00.901309 (+ 3085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:00.901315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:00.901409 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:00.901517 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:00.904448 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.905533 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.908886 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.909644 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.909766 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:00.915596 (+ 5830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:00.915601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:00.915604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:00.915638 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:00.918264 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:00.919273 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:00.919278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:00.919282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:00.919465 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:00.920106 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:00.920114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:00.920120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:00.920445 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:00.920604 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:00.921858 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0926 21:30:00.922111 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:30:00.923707 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.897065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34745 (local address 127.0.97.3:33207)
0926 21:30:00.898020 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:00.898084 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:00.900923 (+ 2839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:00.902121 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:00.902136 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:00.903137 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:00.904211 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:00.904248 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.905710 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.905721 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:00.908662 (+ 2941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:00.908679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.910178 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.910188 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:00.910662 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:00.911627 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:00.911668 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:00.915336 (+ 3668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:00.918416 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:00.918454 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:00.918472 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:00.919133 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:00.919628 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:00.919635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:00.919641 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:00.919913 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:00.920717 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:00.920729 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:00.921267 (+ 538us) client_negotiation.cc:769] Sending connection context
0926 21:30:00.921701 (+ 434us) client_negotiation.cc:240] Negotiation successful
0926 21:30:00.922328 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":448,"threads_started":1}
I20260926 21:30:00.947932 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.933311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46563 (local address 127.0.97.62:34745)
0926 21:30:00.933466 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:00.933470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:00.934144 (+ 674us) server_negotiation.cc:407] Connection header received
0926 21:30:00.934338 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:00.934342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:00.934398 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:00.934560 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:00.935948 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.936998 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.939102 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.939526 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.939669 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:00.941672 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:00.941677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:00.941680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:00.941735 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:00.945235 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:00.945900 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:00.945905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:00.945910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:00.946106 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:00.946482 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:00.946489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:00.946492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:00.946753 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:00.946946 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:00.947525 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 21:30:00.947692 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 21:30:00.949262 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.933391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34745 (local address 127.0.97.3:46563)
0926 21:30:00.934042 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:00.934064 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:00.934214 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:00.934587 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:00.934592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:00.935318 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:00.935795 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:00.935808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.937128 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.937137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:00.938953 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:00.938964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:00.939712 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:00.939718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:00.939878 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:00.940633 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:00.940654 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:00.941478 (+ 824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:00.945379 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:00.945384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:00.945389 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:00.945776 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:00.946215 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:00.946220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:00.946224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:00.946350 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:00.946923 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:00.946929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:00.947203 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 21:30:00.947387 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 21:30:00.948535 (+ 1148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":354,"threads_started":1}
I20260926 21:30:00.957300 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32827
I20260926 21:30:00.957338 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32827 every 8 connection(s)
I20260926 21:30:00.958518 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:00.960631 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34745
I20260926 21:30:00.960837 1996 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:00.961424 1996 heartbeater.cc:511] Master 127.0.97.62:34745 requested a full tablet report, sending...
I20260926 21:30:00.962152 1718 heartbeater.cc:503] Master 127.0.97.62:34745 was elected leader, sending a full tablet report...
I20260926 21:30:00.962831 1510 ts_manager.cc:194] Registered new tserver with Master: c770813c057347c8a6cacd6878213cb7 (127.0.97.3:32827)
I20260926 21:30:00.963891 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33207
I20260926 21:30:00.964764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 1861
I20260926 21:30:00.964952 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:00.965363 388 external_mini_cluster.cc:1428] Setting key 048251599191bf137ba700f3fae8d34d
I20260926 21:30:00.968083 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:30:00 dist-test-slave-87bt krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:30:01 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:01.012661 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:00.999401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.0.97.62:34745)
0926 21:30:00.999640 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:00.999646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:00.999849 (+ 203us) server_negotiation.cc:407] Connection header received
0926 21:30:01.000011 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.000014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.000077 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.000163 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:01.001620 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.002369 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.003664 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.004071 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.004158 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.007754 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:01.007759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:01.007762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:01.007793 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:01.010352 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:01.011000 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:01.011004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:01.011007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:01.011117 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:01.011426 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:01.011429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:01.011431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:01.011689 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:01.011863 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.012106 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.012280 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 21:30:01.016737 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:01.023568 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:01.061264 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.036021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.0.97.1:35077)
0926 21:30:01.053469 (+ 17448us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.053481 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.053515 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:30:01.053619 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.053627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.053886 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.054938 (+ 1052us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:01.055123 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.056577 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.058712 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.060213 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.060345 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.060567 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.060869 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.060998 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17301,"thread_start_us":17190,"threads_started":1}
I20260926 21:30:01.061456 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.035953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35077 (local address 127.0.0.1:43920)
0926 21:30:01.051161 (+ 15208us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.051254 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.051528 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.054112 (+ 2584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.054121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.054170 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:01.054730 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.054742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.056714 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.056720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.058564 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.058575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.060402 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.060408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.060606 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.060659 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.060811 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.061007 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14818,"spinlock_wait_cycles":16512,"thread_start_us":15828,"threads_started":1}
I20260926 21:30:01.062062 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.038907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32827 (local address 127.0.0.1:54822)
0926 21:30:01.053809 (+ 14902us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.053832 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.053959 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.054900 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.054904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.054930 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:01.055321 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.055333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.057283 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.057288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.059426 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.059443 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.061318 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.061326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.061520 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.061554 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.061645 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.061776 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14778,"thread_start_us":13744,"threads_started":1}
I20260926 21:30:01.062722 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.036019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35063 (local address 127.0.0.1:34536)
0926 21:30:01.052940 (+ 16921us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.052967 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.053102 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.053943 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.053950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.053980 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:01.054497 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.054514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.056377 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.056382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.058720 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.058734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.061617 (+ 2883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.061623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.062119 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.062151 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.062230 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.062347 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16791,"mutex_wait_us":2,"spinlock_wait_cycles":5504,"thread_start_us":17532,"threads_started":1}
I20260926 21:30:01.062894 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.039293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.0.97.3:32827)
0926 21:30:01.053025 (+ 13732us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.053038 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.054210 (+ 1172us) server_negotiation.cc:407] Connection header received
0926 21:30:01.054485 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.054493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.054720 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.054926 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:01.055560 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.057130 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.059612 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.061087 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.061228 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.061455 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.062472 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.062609 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13528,"thread_start_us":13340,"threads_started":1}
I20260926 21:30:01.063473 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.036488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.0.97.2:35063)
0926 21:30:01.052257 (+ 15769us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.052269 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.053234 (+ 965us) server_negotiation.cc:407] Connection header received
0926 21:30:01.053364 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.053376 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.053689 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.053949 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:01.054723 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.056205 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.058878 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.060214 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.060345 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.060566 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.063003 (+ 2437us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.063190 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15533,"thread_start_us":15770,"threads_started":1}
I20260926 21:30:01.065635 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 55e0e29e53084e5bb6e73c2ee9804c3a (DEFAULT_TABLE table=test-table [id=0e15fdfe893644e7a9799ddfc4477921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:01.066322 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 55e0e29e53084e5bb6e73c2ee9804c3a (DEFAULT_TABLE table=test-table [id=0e15fdfe893644e7a9799ddfc4477921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:01.067152 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e0e29e53084e5bb6e73c2ee9804c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.067543 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e0e29e53084e5bb6e73c2ee9804c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.067076 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 55e0e29e53084e5bb6e73c2ee9804c3a (DEFAULT_TABLE table=test-table [id=0e15fdfe893644e7a9799ddfc4477921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:01.068306 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e0e29e53084e5bb6e73c2ee9804c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.080243 2015 tablet_bootstrap.cc:492] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Bootstrap starting.
I20260926 21:30:01.084651 2015 tablet_bootstrap.cc:654] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.084923 2016 tablet_bootstrap.cc:492] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Bootstrap starting.
I20260926 21:30:01.085958 2015 log.cc:824] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:01.086534 2017 tablet_bootstrap.cc:492] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Bootstrap starting.
I20260926 21:30:01.088198 2016 tablet_bootstrap.cc:654] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.089208 2015 tablet_bootstrap.cc:492] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: No bootstrap required, opened a new log
I20260926 21:30:01.089242 2017 tablet_bootstrap.cc:654] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.089485 2015 ts_tablet_manager.cc:1397] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 21:30:01.089440 2016 log.cc:824] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:01.090381 2017 log.cc:824] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:01.092559 2016 tablet_bootstrap.cc:492] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: No bootstrap required, opened a new log
I20260926 21:30:01.093333 2016 ts_tablet_manager.cc:1397] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 21:30:01.093978 2017 tablet_bootstrap.cc:492] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: No bootstrap required, opened a new log
I20260926 21:30:01.094249 2017 ts_tablet_manager.cc:1397] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 21:30:01.100839 2015 raft_consensus.cc:348] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.101353 2015 raft_consensus.cc:374] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.101511 2015 raft_consensus.cc:729] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770813c057347c8a6cacd6878213cb7, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.101939 2017 raft_consensus.cc:348] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.102355 2017 raft_consensus.cc:374] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.102210 2015 consensus_queue.cc:260] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.102479 2017 raft_consensus.cc:729] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5619edcf42754ec3b8594c104ce37d6f, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.103760 1996 heartbeater.cc:503] Master 127.0.97.62:34745 was elected leader, sending a full tablet report...
I20260926 21:30:01.104236 2015 ts_tablet_manager.cc:1428] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260926 21:30:01.104521 2016 raft_consensus.cc:348] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.104890 2016 raft_consensus.cc:374] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.104746 2017 consensus_queue.cc:260] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.105006 2016 raft_consensus.cc:729] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa49273c753941d29eee0d2e55ca040c, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.105585 2016 consensus_queue.cc:260] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.106980 1857 heartbeater.cc:503] Master 127.0.97.62:34745 was elected leader, sending a full tablet report...
I20260926 21:30:01.107293 2016 ts_tablet_manager.cc:1428] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260926 21:30:01.108209 2017 ts_tablet_manager.cc:1428] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Time spent starting tablet: real 0.014s user 0.002s sys 0.009s
W20260926 21:30:01.196516 1858 tablet.cc:2369] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:30:01.204386 1719 tablet.cc:2369] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:30:01.210146 1997 tablet.cc:2369] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:01.374933 2021 raft_consensus.cc:482] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:01.375272 2021 raft_consensus.cc:504] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.376933 2021 leader_election.cc:288] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5619edcf42754ec3b8594c104ce37d6f (127.0.97.1:35077), aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063)
I20260926 21:30:01.382839 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.377455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42459 (local address 127.0.97.1:35077)
0926 21:30:01.377663 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.377671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.377698 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:30:01.377772 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.377777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.377861 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.377987 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:01.378400 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.379151 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.381241 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.381978 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.382121 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.382215 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.382488 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.382610 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:30:01.382895 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.377102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35077 (local address 127.0.97.3:42459)
0926 21:30:01.377275 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.377296 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.377435 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.378017 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.378022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.378101 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:01.378258 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.378266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.379286 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.379292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.381077 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.381088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.382131 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.382136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.382312 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.382334 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.382466 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.382598 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1,"spinlock_wait_cycles":21248}
I20260926 21:30:01.383245 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.377376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56419 (local address 127.0.97.2:35063)
0926 21:30:01.377559 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.377566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.377989 (+ 423us) server_negotiation.cc:407] Connection header received
0926 21:30:01.378169 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.378180 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.378232 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.378323 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:01.378858 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.379572 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.381574 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.382340 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.382525 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.382630 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.382888 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.383016 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:30:01.383306 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.377361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35063 (local address 127.0.97.3:56419)
0926 21:30:01.377891 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.377905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.378048 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.378335 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.378340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.378361 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:01.378716 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.378724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.379704 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.379710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.381410 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.381422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.382577 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.382583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.382742 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.382764 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.382864 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.382988 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":317,"threads_started":1}
I20260926 21:30:01.384243 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e0e29e53084e5bb6e73c2ee9804c3a" candidate_uuid: "c770813c057347c8a6cacd6878213cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5619edcf42754ec3b8594c104ce37d6f" is_pre_election: true
I20260926 21:30:01.384248 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e0e29e53084e5bb6e73c2ee9804c3a" candidate_uuid: "c770813c057347c8a6cacd6878213cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49273c753941d29eee0d2e55ca040c" is_pre_election: true
I20260926 21:30:01.384702 1672 raft_consensus.cc:2454] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770813c057347c8a6cacd6878213cb7 in term 0.
I20260926 21:30:01.384721 1807 raft_consensus.cc:2454] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770813c057347c8a6cacd6878213cb7 in term 0.
I20260926 21:30:01.385342 1877 leader_election.cc:302] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5619edcf42754ec3b8594c104ce37d6f, aa49273c753941d29eee0d2e55ca040c, c770813c057347c8a6cacd6878213cb7; no voters:
I20260926 21:30:01.385723 2021 raft_consensus.cc:2781] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:01.385856 2021 raft_consensus.cc:482] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:01.386067 2021 raft_consensus.cc:3037] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.388520 2021 raft_consensus.cc:504] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.389182 2021 leader_election.cc:288] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [CANDIDATE]: Term 1 election: Requested vote from peers 5619edcf42754ec3b8594c104ce37d6f (127.0.97.1:35077), aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063)
I20260926 21:30:01.389657 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e0e29e53084e5bb6e73c2ee9804c3a" candidate_uuid: "c770813c057347c8a6cacd6878213cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5619edcf42754ec3b8594c104ce37d6f"
I20260926 21:30:01.389672 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e0e29e53084e5bb6e73c2ee9804c3a" candidate_uuid: "c770813c057347c8a6cacd6878213cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49273c753941d29eee0d2e55ca040c"
I20260926 21:30:01.389878 1807 raft_consensus.cc:3037] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.389875 1672 raft_consensus.cc:3037] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.393529 1807 raft_consensus.cc:2454] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770813c057347c8a6cacd6878213cb7 in term 1.
I20260926 21:30:01.393529 1672 raft_consensus.cc:2454] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770813c057347c8a6cacd6878213cb7 in term 1.
I20260926 21:30:01.394057 1879 leader_election.cc:302] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5619edcf42754ec3b8594c104ce37d6f, aa49273c753941d29eee0d2e55ca040c, c770813c057347c8a6cacd6878213cb7; no voters:
I20260926 21:30:01.394500 2021 raft_consensus.cc:2781] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:01.394982 2021 raft_consensus.cc:686] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 1 LEADER]: Becoming Leader. State: Replica: c770813c057347c8a6cacd6878213cb7, State: Running, Role: LEADER
I20260926 21:30:01.395459 2021 consensus_queue.cc:237] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.400388 1510 catalog_manager.cc:5495] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c770813c057347c8a6cacd6878213cb7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c770813c057347c8a6cacd6878213cb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } health_report { overall_health: HEALTHY } } }
I20260926 21:30:01.434396 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.429406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.0.97.3:32827)
0926 21:30:01.429603 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.429607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.429735 (+ 128us) server_negotiation.cc:407] Connection header received
0926 21:30:01.430062 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.430065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.430143 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.430263 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:01.430596 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.431280 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.432504 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.432886 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.432982 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.433435 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:01.433939 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:01.434044 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.434135 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.434198 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 21:30:01.442925 1807 raft_consensus.cc:1261] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Refusing update from remote peer c770813c057347c8a6cacd6878213cb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:01.442972 1672 raft_consensus.cc:1261] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Refusing update from remote peer c770813c057347c8a6cacd6878213cb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:01.443846 2025 consensus_queue.cc:1035] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:01.444242 2021 consensus_queue.cc:1035] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:01.480650 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:01.480968 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:01.484826 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31: Sending DeleteTablet for 3 replicas of tablet 55e0e29e53084e5bb6e73c2ee9804c3a
I20260926 21:30:01.486043 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e0e29e53084e5bb6e73c2ee9804c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34536
I20260926 21:30:01.486043 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e0e29e53084e5bb6e73c2ee9804c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43920
I20260926 21:30:01.487147 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e0e29e53084e5bb6e73c2ee9804c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54822
I20260926 21:30:01.487470 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:01.488710 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:01.495811 2039 tablet_replica.cc:330] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: stopping tablet replica
I20260926 21:30:01.496477 2039 raft_consensus.cc:2229] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:01.497313 2039 raft_consensus.cc:2258] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:01.497965 1786 tablet_service.cc:1467] Processing CreateTablet for tablet db41c90b7086472ab22e370684735091 (DEFAULT_TABLE table=test-table [id=7cc7ec7452dc48c49f86fbeaf3bbe748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:01.498586 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db41c90b7086472ab22e370684735091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.499365 1647 tablet_service.cc:1467] Processing CreateTablet for tablet db41c90b7086472ab22e370684735091 (DEFAULT_TABLE table=test-table [id=7cc7ec7452dc48c49f86fbeaf3bbe748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:01.500476 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db41c90b7086472ab22e370684735091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.503854 1926 tablet_service.cc:1467] Processing CreateTablet for tablet db41c90b7086472ab22e370684735091 (DEFAULT_TABLE table=test-table [id=7cc7ec7452dc48c49f86fbeaf3bbe748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:01.504323 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db41c90b7086472ab22e370684735091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.508898 2039 ts_tablet_manager.cc:1905] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:01.509414 2016 tablet_bootstrap.cc:492] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: Bootstrap starting.
I20260926 21:30:01.511080 2016 tablet_bootstrap.cc:654] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.511234 2015 tablet_bootstrap.cc:492] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: Bootstrap starting.
I20260926 21:30:01.515149 2015 tablet_bootstrap.cc:654] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.522362 2016 tablet_bootstrap.cc:492] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: No bootstrap required, opened a new log
I20260926 21:30:01.522639 2016 ts_tablet_manager.cc:1397] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260926 21:30:01.523550 2016 raft_consensus.cc:348] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.523883 2016 raft_consensus.cc:374] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.524029 2016 raft_consensus.cc:729] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa49273c753941d29eee0d2e55ca040c, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.524391 2016 consensus_queue.cc:260] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.528746 2015 tablet_bootstrap.cc:492] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: No bootstrap required, opened a new log
I20260926 21:30:01.529004 2015 ts_tablet_manager.cc:1397] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20260926 21:30:01.529886 2015 raft_consensus.cc:348] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.530231 2015 raft_consensus.cc:374] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.530371 2015 raft_consensus.cc:729] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770813c057347c8a6cacd6878213cb7, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.530736 2015 consensus_queue.cc:260] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.531423 2015 ts_tablet_manager.cc:1428] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:01.532017 2016 ts_tablet_manager.cc:1428] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260926 21:30:01.535410 2017 tablet_bootstrap.cc:492] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: Bootstrap starting.
I20260926 21:30:01.537505 2017 tablet_bootstrap.cc:654] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.538335 2039 ts_tablet_manager.cc:1918] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:01.538692 2039 log.cc:1196] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/55e0e29e53084e5bb6e73c2ee9804c3a
I20260926 21:30:01.539403 2039 ts_tablet_manager.cc:1939] T 55e0e29e53084e5bb6e73c2ee9804c3a P c770813c057347c8a6cacd6878213cb7: Deleting consensus metadata
I20260926 21:30:01.541981 1496 catalog_manager.cc:4841] TS c770813c057347c8a6cacd6878213cb7 (127.0.97.3:32827): tablet 55e0e29e53084e5bb6e73c2ee9804c3a (table test-table [id=0e15fdfe893644e7a9799ddfc4477921]) successfully deleted
I20260926 21:30:01.542392 2037 tablet_replica.cc:330] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: stopping tablet replica
I20260926 21:30:01.542668 2038 tablet_replica.cc:330] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: stopping tablet replica
I20260926 21:30:01.543011 2037 raft_consensus.cc:2229] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:01.543119 2038 raft_consensus.cc:2229] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:01.543550 2038 raft_consensus.cc:2258] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:01.543586 2037 raft_consensus.cc:2258] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:01.545225 2017 tablet_bootstrap.cc:492] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: No bootstrap required, opened a new log
I20260926 21:30:01.545455 2017 ts_tablet_manager.cc:1397] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260926 21:30:01.545583 2038 ts_tablet_manager.cc:1905] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:01.545593 2037 ts_tablet_manager.cc:1905] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:01.546499 2017 raft_consensus.cc:348] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.547087 2017 raft_consensus.cc:374] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.547248 2017 raft_consensus.cc:729] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5619edcf42754ec3b8594c104ce37d6f, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.547642 2017 consensus_queue.cc:260] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.552459 2017 ts_tablet_manager.cc:1428] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260926 21:30:01.553136 2038 ts_tablet_manager.cc:1918] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:01.553243 2037 ts_tablet_manager.cc:1918] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:01.553337 2038 log.cc:1196] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/55e0e29e53084e5bb6e73c2ee9804c3a
I20260926 21:30:01.553395 2037 log.cc:1196] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/55e0e29e53084e5bb6e73c2ee9804c3a
I20260926 21:30:01.553920 2038 ts_tablet_manager.cc:1939] T 55e0e29e53084e5bb6e73c2ee9804c3a P aa49273c753941d29eee0d2e55ca040c: Deleting consensus metadata
I20260926 21:30:01.554090 2037 ts_tablet_manager.cc:1939] T 55e0e29e53084e5bb6e73c2ee9804c3a P 5619edcf42754ec3b8594c104ce37d6f: Deleting consensus metadata
I20260926 21:30:01.556030 1495 catalog_manager.cc:4841] TS 5619edcf42754ec3b8594c104ce37d6f (127.0.97.1:35077): tablet 55e0e29e53084e5bb6e73c2ee9804c3a (table test-table [id=0e15fdfe893644e7a9799ddfc4477921]) successfully deleted
I20260926 21:30:01.556326 1497 catalog_manager.cc:4841] TS aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063): tablet 55e0e29e53084e5bb6e73c2ee9804c3a (table test-table [id=0e15fdfe893644e7a9799ddfc4477921]) successfully deleted
I20260926 21:30:01.660249 2022 raft_consensus.cc:482] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:01.660506 2022 raft_consensus.cc:504] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.662101 2022 leader_election.cc:288] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063), c770813c057347c8a6cacd6878213cb7 (127.0.97.3:32827)
I20260926 21:30:01.670704 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.663093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50325 (local address 127.0.97.2:35063)
0926 21:30:01.663285 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.663291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.663551 (+ 260us) server_negotiation.cc:407] Connection header received
0926 21:30:01.663917 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.663921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.663982 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.664035 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:01.664642 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.665390 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.668091 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.669348 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.669484 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.669598 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.670167 (+ 569us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.670330 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:30:01.671294 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.662652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32827 (local address 127.0.97.1:43519)
0926 21:30:01.663960 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.663980 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.664177 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.664722 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.664727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.664754 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:01.665800 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.665819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.667490 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.667496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.668996 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.669007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.670342 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.670349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.670597 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.670636 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.670742 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.670902 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":954,"threads_started":1}
I20260926 21:30:01.671612 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.662571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35063 (local address 127.0.97.1:50325)
0926 21:30:01.663393 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.663420 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.663704 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.664195 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.664198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.664219 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:01.664484 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.664491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.665544 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.665551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.667901 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.667915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.669766 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.669773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.669959 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.670001 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.671001 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.671153 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":709,"threads_started":1}
I20260926 21:30:01.671929 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.662559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43519 (local address 127.0.97.3:32827)
0926 21:30:01.662873 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.662895 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.664075 (+ 1180us) server_negotiation.cc:407] Connection header received
0926 21:30:01.664363 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.664369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.664451 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.664523 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:01.666047 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.667344 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.669174 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.670135 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.670297 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.670448 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.671046 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.671191 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260926 21:30:01.672358 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db41c90b7086472ab22e370684735091" candidate_uuid: "5619edcf42754ec3b8594c104ce37d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49273c753941d29eee0d2e55ca040c" is_pre_election: true
I20260926 21:30:01.672652 1807 raft_consensus.cc:2454] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5619edcf42754ec3b8594c104ce37d6f in term 0.
I20260926 21:30:01.672722 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db41c90b7086472ab22e370684735091" candidate_uuid: "5619edcf42754ec3b8594c104ce37d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770813c057347c8a6cacd6878213cb7" is_pre_election: true
I20260926 21:30:01.673183 1600 leader_election.cc:302] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5619edcf42754ec3b8594c104ce37d6f, aa49273c753941d29eee0d2e55ca040c; no voters:
I20260926 21:30:01.673169 1946 raft_consensus.cc:2454] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5619edcf42754ec3b8594c104ce37d6f in term 0.
I20260926 21:30:01.673599 2022 raft_consensus.cc:2781] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:01.673746 2022 raft_consensus.cc:482] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:01.673858 2022 raft_consensus.cc:3037] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.676543 2022 raft_consensus.cc:504] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.677258 2022 leader_election.cc:288] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [CANDIDATE]: Term 1 election: Requested vote from peers aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063), c770813c057347c8a6cacd6878213cb7 (127.0.97.3:32827)
I20260926 21:30:01.677763 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db41c90b7086472ab22e370684735091" candidate_uuid: "5619edcf42754ec3b8594c104ce37d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49273c753941d29eee0d2e55ca040c"
I20260926 21:30:01.677899 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db41c90b7086472ab22e370684735091" candidate_uuid: "5619edcf42754ec3b8594c104ce37d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770813c057347c8a6cacd6878213cb7"
I20260926 21:30:01.677982 1807 raft_consensus.cc:3037] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.678102 1946 raft_consensus.cc:3037] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.681128 1807 raft_consensus.cc:2454] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5619edcf42754ec3b8594c104ce37d6f in term 1.
I20260926 21:30:01.681420 1946 raft_consensus.cc:2454] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5619edcf42754ec3b8594c104ce37d6f in term 1.
I20260926 21:30:01.681823 1600 leader_election.cc:302] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5619edcf42754ec3b8594c104ce37d6f, aa49273c753941d29eee0d2e55ca040c; no voters:
I20260926 21:30:01.682195 2022 raft_consensus.cc:2781] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:01.682780 2022 raft_consensus.cc:686] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 1 LEADER]: Becoming Leader. State: Replica: 5619edcf42754ec3b8594c104ce37d6f, State: Running, Role: LEADER
I20260926 21:30:01.683189 2022 consensus_queue.cc:237] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } }
I20260926 21:30:01.687027 1509 catalog_manager.cc:5495] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5619edcf42754ec3b8594c104ce37d6f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } health_report { overall_health: UNKNOWN } } }
Sep 26 21:30:01 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:01.743580 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.729044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.0.97.62:34745)
0926 21:30:01.730068 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.730076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.730100 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:30:01.730151 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.730154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.730203 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.730321 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:01.731491 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.732281 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.733483 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.733915 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.734039 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.737565 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:01.737569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:01.737573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:01.737599 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:01.740728 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:01.741431 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:01.741439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:01.741443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:01.741567 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:01.741934 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:01.741940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:01.741946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:01.742205 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:01.742363 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.742847 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.743175 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":908,"thread_start_us":766,"threads_started":1}
I20260926 21:30:01.743603 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.729032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34745 (local address 127.0.0.1:48036)
0926 21:30:01.729787 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.729812 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.729997 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.730345 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.730349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.731107 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:01.731317 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.731327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.732410 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.732418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.733337 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.733354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.734084 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.734089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.734268 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.735026 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:01.735043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:01.737366 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:01.740879 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:01.740895 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:01.740903 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:01.741317 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:01.741679 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:01.741684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:01.741687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:01.741831 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:01.742338 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:01.742346 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:01.742567 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.742748 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.743049 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":417,"threads_started":1}
I20260926 21:30:01.749505 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48036:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 21:30:01.750648 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:01.758111 1926 tablet_service.cc:1467] Processing CreateTablet for tablet d60aae29a97c4c84b00dc1fa1b431304 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06839a56905b42d8b269ecc201f548f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:01.758093 1648 tablet_service.cc:1467] Processing CreateTablet for tablet d60aae29a97c4c84b00dc1fa1b431304 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06839a56905b42d8b269ecc201f548f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:01.758761 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60aae29a97c4c84b00dc1fa1b431304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.758760 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60aae29a97c4c84b00dc1fa1b431304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.762988 1787 tablet_service.cc:1467] Processing CreateTablet for tablet d60aae29a97c4c84b00dc1fa1b431304 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06839a56905b42d8b269ecc201f548f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:01.763579 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60aae29a97c4c84b00dc1fa1b431304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:01.764586 2017 tablet_bootstrap.cc:492] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: Bootstrap starting.
I20260926 21:30:01.766259 2017 tablet_bootstrap.cc:654] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.767155 2015 tablet_bootstrap.cc:492] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: Bootstrap starting.
I20260926 21:30:01.768991 2015 tablet_bootstrap.cc:654] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.772230 2016 tablet_bootstrap.cc:492] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: Bootstrap starting.
I20260926 21:30:01.774652 2016 tablet_bootstrap.cc:654] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:01.779155 2017 tablet_bootstrap.cc:492] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: No bootstrap required, opened a new log
I20260926 21:30:01.779379 2017 ts_tablet_manager.cc:1397] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260926 21:30:01.780769 2017 raft_consensus.cc:348] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.781044 2017 raft_consensus.cc:374] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.781137 2017 raft_consensus.cc:729] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5619edcf42754ec3b8594c104ce37d6f, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.781437 2017 consensus_queue.cc:260] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.781610 2016 tablet_bootstrap.cc:492] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: No bootstrap required, opened a new log
I20260926 21:30:01.781898 2022 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.782169 2022 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:01.782511 2015 tablet_bootstrap.cc:492] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: No bootstrap required, opened a new log
I20260926 21:30:01.782737 2015 ts_tablet_manager.cc:1397] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260926 21:30:01.781774 2016 ts_tablet_manager.cc:1397] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260926 21:30:01.783748 2016 raft_consensus.cc:348] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.783730 2015 raft_consensus.cc:348] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.783996 2016 raft_consensus.cc:374] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.784085 2016 raft_consensus.cc:729] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa49273c753941d29eee0d2e55ca040c, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.784137 2015 raft_consensus.cc:374] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:01.784323 2015 raft_consensus.cc:729] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770813c057347c8a6cacd6878213cb7, State: Initialized, Role: FOLLOWER
I20260926 21:30:01.784404 2016 consensus_queue.cc:260] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.784780 2015 consensus_queue.cc:260] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.785097 2017 ts_tablet_manager.cc:1428] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 21:30:01.784862 2023 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.785228 2023 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:01.785426 2021 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.785635 2015 ts_tablet_manager.cc:1428] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:01.785706 2021 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:01.786563 2016 ts_tablet_manager.cc:1428] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 21:30:01.821744 2022 raft_consensus.cc:482] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:01.821985 2022 raft_consensus.cc:504] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.822806 2022 leader_election.cc:288] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770813c057347c8a6cacd6878213cb7 (127.0.97.3:32827), aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063)
I20260926 21:30:01.823344 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60aae29a97c4c84b00dc1fa1b431304" candidate_uuid: "5619edcf42754ec3b8594c104ce37d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770813c057347c8a6cacd6878213cb7" is_pre_election: true
I20260926 21:30:01.823385 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60aae29a97c4c84b00dc1fa1b431304" candidate_uuid: "5619edcf42754ec3b8594c104ce37d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49273c753941d29eee0d2e55ca040c" is_pre_election: true
I20260926 21:30:01.823603 1946 raft_consensus.cc:2454] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5619edcf42754ec3b8594c104ce37d6f in term 0.
I20260926 21:30:01.823660 1807 raft_consensus.cc:2454] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5619edcf42754ec3b8594c104ce37d6f in term 0.
I20260926 21:30:01.824062 1599 leader_election.cc:302] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5619edcf42754ec3b8594c104ce37d6f, c770813c057347c8a6cacd6878213cb7; no voters:
I20260926 21:30:01.824808 2022 raft_consensus.cc:2781] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:01.824918 2022 raft_consensus.cc:482] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:01.825058 2022 raft_consensus.cc:3037] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.827322 2022 raft_consensus.cc:504] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.827941 2022 leader_election.cc:288] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [CANDIDATE]: Term 1 election: Requested vote from peers c770813c057347c8a6cacd6878213cb7 (127.0.97.3:32827), aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063)
I20260926 21:30:01.828473 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60aae29a97c4c84b00dc1fa1b431304" candidate_uuid: "5619edcf42754ec3b8594c104ce37d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770813c057347c8a6cacd6878213cb7"
I20260926 21:30:01.828438 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60aae29a97c4c84b00dc1fa1b431304" candidate_uuid: "5619edcf42754ec3b8594c104ce37d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49273c753941d29eee0d2e55ca040c"
I20260926 21:30:01.828708 1946 raft_consensus.cc:3037] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.828709 1807 raft_consensus.cc:3037] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:01.831719 1807 raft_consensus.cc:2454] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5619edcf42754ec3b8594c104ce37d6f in term 1.
I20260926 21:30:01.831907 1946 raft_consensus.cc:2454] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5619edcf42754ec3b8594c104ce37d6f in term 1.
I20260926 21:30:01.832299 1600 leader_election.cc:302] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5619edcf42754ec3b8594c104ce37d6f, aa49273c753941d29eee0d2e55ca040c; no voters:
I20260926 21:30:01.832762 2022 raft_consensus.cc:2781] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:01.832935 2022 raft_consensus.cc:686] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [term 1 LEADER]: Becoming Leader. State: Replica: 5619edcf42754ec3b8594c104ce37d6f, State: Running, Role: LEADER
I20260926 21:30:01.833250 2022 consensus_queue.cc:237] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } }
I20260926 21:30:01.835698 2045 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: TxnStatusTablet state changed. Reason: New leader 5619edcf42754ec3b8594c104ce37d6f. Latest consensus state: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.836083 2045 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:01.836879 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 21:30:01.837114 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 21:30:01.838150 1510 catalog_manager.cc:5495] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5619edcf42754ec3b8594c104ce37d6f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } health_report { overall_health: UNKNOWN } } }
I20260926 21:30:01.869421 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.861003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.97.1:35077)
0926 21:30:01.861234 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.861241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.861473 (+ 232us) server_negotiation.cc:407] Connection header received
0926 21:30:01.861809 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.861816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.861928 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.862099 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:01.862561 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.864240 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.866304 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.867006 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.867146 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.867796 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:01.868666 (+ 870us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:01.868832 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.868955 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.869055 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 21:30:01.869463 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.861130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35077 (local address 127.0.0.1:43922)
0926 21:30:01.861353 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.861376 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.861552 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.862158 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.862163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.862178 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:01.862379 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.862392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.864425 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.864432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.866092 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.866116 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.867216 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.867223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.867472 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.867565 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:01.868805 (+ 1240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:01.868831 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.868929 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.869106 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 21:30:01.875452 1807 raft_consensus.cc:1261] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Refusing update from remote peer 5619edcf42754ec3b8594c104ce37d6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:01.875620 1946 raft_consensus.cc:1261] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Refusing update from remote peer 5619edcf42754ec3b8594c104ce37d6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:01.876566 2045 consensus_queue.cc:1035] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:01.877171 2022 consensus_queue.cc:1035] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:01.888237 2023 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: TxnStatusTablet state changed. Reason: New leader 5619edcf42754ec3b8594c104ce37d6f. Latest consensus state: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.888607 2023 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:01.890480 2021 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: TxnStatusTablet state changed. Reason: New leader 5619edcf42754ec3b8594c104ce37d6f. Latest consensus state: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.890784 2021 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:01.891193 2022 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.891525 2022 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:01.891592 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333716794871869440 to 7333716794713423872. Current Snapshot: MvccSnapshot[applied={T|T < 7333716794871869440}]
I20260926 21:30:01.897127 2021 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.897482 2021 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P c770813c057347c8a6cacd6878213cb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:01.899533 2045 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.899807 2045 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P 5619edcf42754ec3b8594c104ce37d6f: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:01.905498 2023 tablet_replica.cc:439] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5619edcf42754ec3b8594c104ce37d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5619edcf42754ec3b8594c104ce37d6f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35077 } } peers { permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 } } peers { permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 } } }
I20260926 21:30:01.905759 2023 tablet_replica.cc:442] T d60aae29a97c4c84b00dc1fa1b431304 P aa49273c753941d29eee0d2e55ca040c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:01.912320 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.904040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.0.97.1:35077)
0926 21:30:01.904225 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.904231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.904415 (+ 184us) server_negotiation.cc:407] Connection header received
0926 21:30:01.904630 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.904636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.904744 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.906670 (+ 1926us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:01.906823 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.908019 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.910129 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.910847 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.910980 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.911519 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:01.911652 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:01.911755 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.911936 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.912016 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:30:01.942502 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.925223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48025 (local address 127.0.97.1:35077)
0926 21:30:01.925436 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:01.925442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:01.935183 (+ 9741us) server_negotiation.cc:407] Connection header received
0926 21:30:01.935404 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:01.935410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:01.935498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:01.935615 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:01.936250 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.937474 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.940190 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.940888 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.941031 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.941592 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:01.941749 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:01.941840 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:01.942001 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 21:30:01.942080 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:30:01.942864 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:01.925099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35077 (local address 127.0.97.1:48025)
0926 21:30:01.935023 (+ 9924us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:01.935047 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:01.935249 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:01.935643 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:01.935650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:01.935664 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:01.936065 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.936077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.937618 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.937623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.940021 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:01.940037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:01.941107 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:01.941115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:01.941314 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:01.941399 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:01.941864 (+ 465us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:01.941875 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 21:30:01.941967 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 21:30:01.942125 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9703,"thread_start_us":9530,"threads_started":1}
I20260926 21:30:01.960130 1807 raft_consensus.cc:1261] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Refusing update from remote peer 5619edcf42754ec3b8594c104ce37d6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:01.961040 2045 consensus_queue.cc:1035] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa49273c753941d29eee0d2e55ca040c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:01.966529 1945 raft_consensus.cc:1261] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Refusing update from remote peer 5619edcf42754ec3b8594c104ce37d6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:01.967279 2045 consensus_queue.cc:1035] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770813c057347c8a6cacd6878213cb7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:01.987479 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333716795219693568 to 7333716794098675712. Current Snapshot: MvccSnapshot[applied={T|T < 7333716795219693568}]
I20260926 21:30:02.003327 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333716795219693568 to 7333716794098675712. Current Snapshot: MvccSnapshot[applied={T|T < 7333716795219693568}]
I20260926 21:30:02.058857 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:02.059157 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:02.062355 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c765742b69a40e89838223f74da4f31: Sending DeleteTablet for 3 replicas of tablet db41c90b7086472ab22e370684735091
I20260926 21:30:02.063270 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet db41c90b7086472ab22e370684735091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43920
I20260926 21:30:02.063302 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet db41c90b7086472ab22e370684735091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34536
I20260926 21:30:02.063656 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet db41c90b7086472ab22e370684735091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54822
I20260926 21:30:02.072993 2072 tablet_replica.cc:330] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: stopping tablet replica
I20260926 21:30:02.074041 2072 raft_consensus.cc:2229] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:02.074919 2072 raft_consensus.cc:2258] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:02.077757 2072 ts_tablet_manager.cc:1905] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:02.078896 2071 tablet_replica.cc:330] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: stopping tablet replica
I20260926 21:30:02.079174 2071 raft_consensus.cc:2229] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:02.079797 2071 raft_consensus.cc:2258] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:02.080951 2071 ts_tablet_manager.cc:1905] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:02.087682 2070 tablet_replica.cc:330] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: stopping tablet replica
I20260926 21:30:02.088754 2072 ts_tablet_manager.cc:1918] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:02.088943 2072 log.cc:1196] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/db41c90b7086472ab22e370684735091
I20260926 21:30:02.089452 2072 ts_tablet_manager.cc:1939] T db41c90b7086472ab22e370684735091 P c770813c057347c8a6cacd6878213cb7: Deleting consensus metadata
I20260926 21:30:02.090499 2071 ts_tablet_manager.cc:1918] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:02.090690 2071 log.cc:1196] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/db41c90b7086472ab22e370684735091
I20260926 21:30:02.090652 1496 catalog_manager.cc:4841] TS c770813c057347c8a6cacd6878213cb7 (127.0.97.3:32827): tablet db41c90b7086472ab22e370684735091 (table test-table [id=7cc7ec7452dc48c49f86fbeaf3bbe748]) successfully deleted
I20260926 21:30:02.091164 2071 ts_tablet_manager.cc:1939] T db41c90b7086472ab22e370684735091 P 5619edcf42754ec3b8594c104ce37d6f: Deleting consensus metadata
I20260926 21:30:02.088323 2070 raft_consensus.cc:2229] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:02.092259 1495 catalog_manager.cc:4841] TS 5619edcf42754ec3b8594c104ce37d6f (127.0.97.1:35077): tablet db41c90b7086472ab22e370684735091 (table test-table [id=7cc7ec7452dc48c49f86fbeaf3bbe748]) successfully deleted
I20260926 21:30:02.092324 2070 raft_consensus.cc:2258] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:02.093454 2070 ts_tablet_manager.cc:1905] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:30:02 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:02.100648 2070 ts_tablet_manager.cc:1918] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:02.100934 2070 log.cc:1196] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/db41c90b7086472ab22e370684735091
I20260926 21:30:02.101372 2070 ts_tablet_manager.cc:1939] T db41c90b7086472ab22e370684735091 P aa49273c753941d29eee0d2e55ca040c: Deleting consensus metadata
I20260926 21:30:02.102625 1497 catalog_manager.cc:4841] TS aa49273c753941d29eee0d2e55ca040c (127.0.97.2:35063): tablet db41c90b7086472ab22e370684735091 (table test-table [id=7cc7ec7452dc48c49f86fbeaf3bbe748]) successfully deleted
I20260926 21:30:02.112639 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:02.077765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.0.97.1:35077)
0926 21:30:02.077968 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:02.077974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:02.088230 (+ 10256us) server_negotiation.cc:407] Connection header received
0926 21:30:02.088445 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:02.088451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:02.088548 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:02.088675 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:02.091833 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.092862 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.094369 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.094876 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.095020 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:02.099678 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:02.099693 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:02.099708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:02.099757 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:02.109610 (+ 9853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.110413 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.110476 (+ 63us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.110483 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.110643 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.111037 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.111040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.111043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.111384 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:02.111726 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:02.112012 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 21:30:02.112243 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 21:30:02.113528 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43940
I20260926 21:30:02.131196 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:02.116588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.0.97.62:34745)
0926 21:30:02.116782 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:02.116789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:02.117213 (+ 424us) server_negotiation.cc:407] Connection header received
0926 21:30:02.117388 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:02.117392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:02.117455 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:02.117529 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:02.119582 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.120350 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.122195 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.122840 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.122948 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:02.125676 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:02.125680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:02.125684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:02.125717 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:02.128743 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.129490 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.129494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.129496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.129608 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.129968 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.129971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.129973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.130221 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:02.130392 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:02.130760 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 21:30:02.130935 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 21:30:02.131759 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48050
I20260926 21:30:02.134842 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1582
I20260926 21:30:02.149053 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:02.135670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.0.97.1:35077)
0926 21:30:02.135873 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:02.135891 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:02.136128 (+ 237us) server_negotiation.cc:407] Connection header received
0926 21:30:02.136268 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:02.136272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:02.136333 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:02.136417 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:02.137845 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.138661 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.140396 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.140864 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.140979 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:02.143729 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:02.143738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:02.143746 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:02.143791 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:02.146625 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.147350 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.147354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.147356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.147476 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.147835 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.147838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.147840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.148074 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:02.148259 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:02.148558 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 21:30:02.148717 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 21:30:02.149652 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43942
W20260926 21:30:02.150556 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:02.150741 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 1582
I20260926 21:30:02.165637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1722
Sep 26 21:30:02 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:30:02.180356 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:02.166429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.0.97.2:35063)
0926 21:30:02.166616 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:02.166621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:02.166639 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:02.166806 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:02.166810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:02.166868 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:02.166955 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:02.168021 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.168714 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.169845 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.170216 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.170300 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:02.174413 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:02.174481 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:02.174502 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:02.174547 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:02.177438 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.178203 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.178210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.178213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.178335 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.178977 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.178980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.178982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.179317 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:02.179552 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:02.179845 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 21:30:02.180012 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 21:30:02.181070 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34538
W20260926 21:30:02.181743 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:02.181929 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 1722
I20260926 21:30:02.198599 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1861
Sep 26 21:30:02 dist-test-slave-87bt krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:30:02.213201 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:02.199413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.0.97.3:32827)
0926 21:30:02.199856 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:02.199861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:02.199878 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:30:02.199932 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:02.199936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:02.200000 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:02.200100 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:02.201335 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.202168 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.203348 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:02.203803 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:02.203901 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:02.207854 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:02.207868 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:02.207885 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:02.207918 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:02.210798 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.211388 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.211396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.211398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.211539 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:02.211865 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:02.211869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:02.211872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:02.212241 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:02.212459 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:02.212687 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 21:30:02.212854 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
W20260926 21:30:02.213979 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54846
W20260926 21:30:02.214733 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:02.214947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 1861
I20260926 21:30:02.236120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 1479
I20260926 21:30:02.237334 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:02.362865 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 1479
2026-09-26T21:30:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4701 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:02 dist-test-slave-87bt krb5kdc[2082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:02 dist-test-slave-87bt krb5kdc[2082](info): set up 2 sockets
Sep 26 21:30:02 dist-test-slave-87bt krb5kdc[2082](info): commencing operation
krb5kdc: starting...
W20260926 21:30:03.450462 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:03 dist-test-slave-87bt krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:03.637728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36205
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:03.961773 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:03.962129 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:03.962299 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:03.969784 2095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:03.969937 2095 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 21:30:03.970032 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:03.970149 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:03.970233 2095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:03.970323 2095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:03.980249 2095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36205
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:03.981573 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:03.982977 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:03.994670 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:03.994772 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:03.994817 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:03.995579 2095 server_base.cc:1034] running on GCE node
I20260926 21:30:03.996412 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:03.997843 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:03.999135 2095 hybrid_clock.cc:648] HybridClock initialized: now 1790458203999060 us; error 46 us; skew 500 ppm
I20260926 21:30:03.999492 2095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:03 dist-test-slave-87bt krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:04.002835 2095 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:30:04.004300 2095 webserver.cc:460] Webserver started at http://127.0.97.62:39615/ using document root <none> and password file <none>
I20260926 21:30:04.005000 2095 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:04.005142 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:04.005407 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:04.008280 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "9361219265d24bed8c606793953f2182"
format_stamp: "Formatted at 2026-09-26 21:30:03 on dist-test-slave-87bt"
server_key: "df48458fe32466b5d632aa449ec8ccfc"
server_key_iv: "09077f8bf9945d53eedb4b0a98966a2e"
server_key_version: "encryptionkey@0"
I20260926 21:30:04.008975 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "9361219265d24bed8c606793953f2182"
format_stamp: "Formatted at 2026-09-26 21:30:03 on dist-test-slave-87bt"
server_key: "df48458fe32466b5d632aa449ec8ccfc"
server_key_iv: "09077f8bf9945d53eedb4b0a98966a2e"
server_key_version: "encryptionkey@0"
I20260926 21:30:04.013983 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:30:04.017526 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:04.019024 2095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:04.019214 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "9361219265d24bed8c606793953f2182"
format_stamp: "Formatted at 2026-09-26 21:30:03 on dist-test-slave-87bt"
server_key: "df48458fe32466b5d632aa449ec8ccfc"
server_key_iv: "09077f8bf9945d53eedb4b0a98966a2e"
server_key_version: "encryptionkey@0"
I20260926 21:30:04.019439 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:04.033972 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:04.038045 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:04.038358 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:04.059139 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36205
I20260926 21:30:04.059161 2171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36205 every 8 connection(s)
I20260926 21:30:04.060560 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:04.064486 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:04.070950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 2095
I20260926 21:30:04.071154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:04.071784 388 external_mini_cluster.cc:1428] Setting key f5626fa5c90e4c9ffc18806eb4e2e6d6
I20260926 21:30:04.074937 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: Bootstrap starting.
I20260926 21:30:04.078378 2172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:04.079563 2172 log.cc:824] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:04.082751 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: No bootstrap required, opened a new log
Sep 26 21:30:04 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:04.091090 2172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9361219265d24bed8c606793953f2182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36205 } }
I20260926 21:30:04.091513 2172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:04.091619 2172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9361219265d24bed8c606793953f2182, State: Initialized, Role: FOLLOWER
I20260926 21:30:04.092288 2172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9361219265d24bed8c606793953f2182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36205 } }
I20260926 21:30:04.092573 2172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:04.092728 2172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:04.092926 2172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:04.096099 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:04.073779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.0.97.62:36205)
0926 21:30:04.074597 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:04.074609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:04.074649 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:30:04.076408 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:04.076430 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:04.076817 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:04.077142 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:04.078958 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.080625 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.082348 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.083146 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.083343 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:04.087656 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:04.087665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:04.087692 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:04.087739 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:04.091235 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:04.092099 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:04.092108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:04.092117 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:04.092302 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:04.092770 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:04.092777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:04.092782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:04.093530 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:04.093823 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:04.094237 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 21:30:04.094732 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":375,"threads_started":1}
I20260926 21:30:04.096223 2172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9361219265d24bed8c606793953f2182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36205 } }
I20260926 21:30:04.096818 2172 leader_election.cc:302] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9361219265d24bed8c606793953f2182; no voters:
I20260926 21:30:04.097473 2172 leader_election.cc:288] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:04.097582 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:04.099560 2172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:04.102700 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [term 1 LEADER]: Becoming Leader. State: Replica: 9361219265d24bed8c606793953f2182, State: Running, Role: LEADER
I20260926 21:30:04.103221 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9361219265d24bed8c606793953f2182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36205 } }
I20260926 21:30:04.106293 2178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9361219265d24bed8c606793953f2182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9361219265d24bed8c606793953f2182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36205 } } }
I20260926 21:30:04.106892 2178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:04.106318 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9361219265d24bed8c606793953f2182. Latest consensus state: current_term: 1 leader_uuid: "9361219265d24bed8c606793953f2182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9361219265d24bed8c606793953f2182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36205 } } }
I20260926 21:30:04.107376 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182 [sys.catalog]: This master's current role is: LEADER
W20260926 21:30:04.113116 2191 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:30:04.113310 2191 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:30:04.113513 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:04.115643 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:04.123932 2192 catalog_manager.cc:1347] Generated new cluster ID: c7994fd15eed46c4bef46c8694f0d226
I20260926 21:30:04.124176 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:04.135629 2192 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:04.137547 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:04.149674 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: Generated new TSK 0
I20260926 21:30:04.150943 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:30:04.199779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36205
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:04.526077 2198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:04.526459 2198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:04.526751 2198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:04.534569 2198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:04.534744 2198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:04.534890 2198 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 21:30:04.534976 2198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:30:04.535102 2198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:04.545169 2198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:04.546671 2198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:04.548205 2198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:04.562127 2203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:04.562158 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:04.562086 2204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:04.563361 2198 server_base.cc:1034] running on GCE node
I20260926 21:30:04.564092 2198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:04.565213 2198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:04.566462 2198 hybrid_clock.cc:648] HybridClock initialized: now 1790458204566355 us; error 67 us; skew 500 ppm
I20260926 21:30:04.566835 2198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:04 dist-test-slave-87bt krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:04.573768 2198 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:30:04.575344 2198 webserver.cc:460] Webserver started at http://127.0.97.1:42371/ using document root <none> and password file <none>
I20260926 21:30:04.576042 2198 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:04.576146 2198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:04.576406 2198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:04.579494 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "3d92fcf19cf1445985a9f25f7bb826d6"
format_stamp: "Formatted at 2026-09-26 21:30:04 on dist-test-slave-87bt"
server_key: "51f75d2fbaf3bd252f6031d3de58a748"
server_key_iv: "b41a4304289b9570edd48db0b4aa3aa4"
server_key_version: "encryptionkey@0"
I20260926 21:30:04.580358 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3d92fcf19cf1445985a9f25f7bb826d6"
format_stamp: "Formatted at 2026-09-26 21:30:04 on dist-test-slave-87bt"
server_key: "51f75d2fbaf3bd252f6031d3de58a748"
server_key_iv: "b41a4304289b9570edd48db0b4aa3aa4"
server_key_version: "encryptionkey@0"
I20260926 21:30:04.585800 2198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:30:04.589784 2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:04.591594 2198 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 21:30:04.591820 2198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "3d92fcf19cf1445985a9f25f7bb826d6"
format_stamp: "Formatted at 2026-09-26 21:30:04 on dist-test-slave-87bt"
server_key: "51f75d2fbaf3bd252f6031d3de58a748"
server_key_iv: "b41a4304289b9570edd48db0b4aa3aa4"
server_key_version: "encryptionkey@0"
I20260926 21:30:04.592032 2198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:04.611018 2198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:04.615917 2198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:04.616241 2198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:04.619046 2198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:04.619153 2198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:04.619343 2198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:04.619442 2198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:30:04 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:04.646903 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:04.620028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60225 (local address 127.0.97.62:36205)
0926 21:30:04.620505 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:04.620511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:04.621876 (+ 1365us) server_negotiation.cc:407] Connection header received
0926 21:30:04.624847 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:04.624852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:04.624946 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:04.625088 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:04.627912 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.629446 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.632001 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.632667 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.632852 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:04.639083 (+ 6231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:04.639094 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:04.639100 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:04.639153 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:04.642750 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:04.643890 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:04.643899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:04.643906 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:04.644127 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:04.644778 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:04.644783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:04.644785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:04.645070 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:04.645233 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:04.646238 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0926 21:30:04.646516 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260926 21:30:04.648571 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:04.620741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36205 (local address 127.0.97.1:60225)
0926 21:30:04.621658 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:04.621721 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:04.624468 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:04.625565 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:04.625607 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:04.626565 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:04.627565 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:04.627606 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.629679 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.629689 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:04.631798 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:04.631814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.633017 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.633029 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:04.633442 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:04.634639 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:04.634684 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:04.638655 (+ 3971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:04.642987 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:04.643001 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:04.643033 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:04.643676 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:04.644284 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:04.644288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:04.644290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:04.644518 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:04.645231 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:04.645241 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:04.645786 (+ 545us) client_negotiation.cc:769] Sending connection context
0926 21:30:04.646165 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 21:30:04.646836 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":389,"threads_started":1}
I20260926 21:30:04.663069 2198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38461
I20260926 21:30:04.663110 2332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38461 every 8 connection(s)
I20260926 21:30:04.664489 2198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:04.666998 2333 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36205
I20260926 21:30:04.667309 2333 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:04.668116 2333 heartbeater.cc:511] Master 127.0.97.62:36205 requested a full tablet report, sending...
I20260926 21:30:04.669845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 2198
I20260926 21:30:04.670025 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:04.670416 2127 ts_manager.cc:194] Registered new tserver with Master: 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461)
I20260926 21:30:04.670827 388 external_mini_cluster.cc:1428] Setting key 7bdd770590d9970f054a1bf9f4728d62
I20260926 21:30:04.673075 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60225
I20260926 21:30:04.679507 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:04.664796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36205 (local address 127.0.97.1:40283)
0926 21:30:04.665793 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:04.665821 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:04.666032 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:04.666917 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:04.666922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:04.667445 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:04.667823 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:04.667844 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.669174 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.669178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:04.670166 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:04.670176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.671081 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.671087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:04.671282 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:04.671955 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:04.671977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:04.672926 (+ 949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:04.677081 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:04.677086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:04.677088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:04.677511 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:04.677963 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:04.677967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:04.677970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:04.678096 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:04.678719 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:04.678723 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:04.678863 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 21:30:04.678976 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 21:30:04.679198 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":491,"threads_started":1}
I20260926 21:30:04.679744 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:04.664811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40283 (local address 127.0.97.62:36205)
0926 21:30:04.665012 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:04.665020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:04.665957 (+ 937us) server_negotiation.cc:407] Connection header received
0926 21:30:04.666199 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:04.666204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:04.666290 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:04.666397 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:04.667982 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.669055 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.670322 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:04.670900 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:04.671029 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:04.673158 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:04.673164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:04.673178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:04.673227 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:04.676910 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:04.677647 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:04.677655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:04.677662 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:04.677828 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:04.678214 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:04.678221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:04.678227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:04.678611 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:04.678774 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:04.679095 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 21:30:04.679352 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:30:04.714368 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36205
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:05.040707 2338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:05.041013 2338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:05.041199 2338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:05.048679 2338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:05.048801 2338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:05.048959 2338 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 21:30:05.049042 2338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:30:05.049186 2338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:05.059474 2338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:05.060751 2338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:05.062073 2338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:05.075326 2344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:05.075367 2346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:05.075785 2343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:05.076217 2338 server_base.cc:1034] running on GCE node
I20260926 21:30:05.076753 2338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:05.077663 2338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:05.078858 2338 hybrid_clock.cc:648] HybridClock initialized: now 1790458205078807 us; error 40 us; skew 500 ppm
I20260926 21:30:05.079190 2338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:05 dist-test-slave-87bt krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:05.082160 2338 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:30:05.083673 2338 webserver.cc:460] Webserver started at http://127.0.97.2:36797/ using document root <none> and password file <none>
I20260926 21:30:05.084364 2338 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:05.084515 2338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:05.084823 2338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:05.087745 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "19bdd83ce79848918c37c032ecd8fffa"
format_stamp: "Formatted at 2026-09-26 21:30:05 on dist-test-slave-87bt"
server_key: "846da6c0cede2462f852b3cf3ac8eadf"
server_key_iv: "11dd5fe5c15fe04a59ba4d79e3b1d127"
server_key_version: "encryptionkey@0"
I20260926 21:30:05.088532 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "19bdd83ce79848918c37c032ecd8fffa"
format_stamp: "Formatted at 2026-09-26 21:30:05 on dist-test-slave-87bt"
server_key: "846da6c0cede2462f852b3cf3ac8eadf"
server_key_iv: "11dd5fe5c15fe04a59ba4d79e3b1d127"
server_key_version: "encryptionkey@0"
I20260926 21:30:05.093742 2338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:30:05.097564 2353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:05.099040 2338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:05.099257 2338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "19bdd83ce79848918c37c032ecd8fffa"
format_stamp: "Formatted at 2026-09-26 21:30:05 on dist-test-slave-87bt"
server_key: "846da6c0cede2462f852b3cf3ac8eadf"
server_key_iv: "11dd5fe5c15fe04a59ba4d79e3b1d127"
server_key_version: "encryptionkey@0"
I20260926 21:30:05.099468 2338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:05.114255 2338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:05.118572 2338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:05.118855 2338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:05.121941 2338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:05.122089 2338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:05.122220 2338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:05.122320 2338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:30:05 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:05.147989 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.122570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41577 (local address 127.0.97.62:36205)
0926 21:30:05.122763 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:05.122769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:05.124287 (+ 1518us) server_negotiation.cc:407] Connection header received
0926 21:30:05.127510 (+ 3223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:05.127517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:05.127617 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:05.127710 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:05.130957 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.132062 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.134577 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.135169 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.135328 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.140760 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:05.140765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:05.140768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:05.140801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:05.143946 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.145121 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.145128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.145131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.145306 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.145901 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.145907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.145912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.146249 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:05.146406 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:05.147392 (+ 986us) server_negotiation.cc:299] Negotiation successful
0926 21:30:05.147637 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 21:30:05.149103 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.123110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36205 (local address 127.0.97.2:41577)
0926 21:30:05.124083 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:05.124142 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:05.127096 (+ 2954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:05.128401 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:05.128418 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:05.129490 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:05.130672 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.130711 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.132293 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.132297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.134374 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.134386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.135459 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.135474 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.135823 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.136898 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:05.136938 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:05.140478 (+ 3540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:05.144168 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:05.144181 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:05.144207 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:05.144936 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:05.145466 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:05.145469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:05.145471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:05.145700 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:05.146385 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:05.146393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:05.146926 (+ 533us) client_negotiation.cc:769] Sending connection context
0926 21:30:05.147294 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 21:30:05.147887 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":325,"threads_started":1}
I20260926 21:30:05.168203 2338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33179
I20260926 21:30:05.168303 2472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33179 every 8 connection(s)
I20260926 21:30:05.169705 2338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:05.172081 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 2338
I20260926 21:30:05.172017 2473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36205
I20260926 21:30:05.172260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:05.172369 2473 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:05.172681 388 external_mini_cluster.cc:1428] Setting key ae478ceae4f40e48d27899e510e2c0f5
I20260926 21:30:05.173240 2473 heartbeater.cc:511] Master 127.0.97.62:36205 requested a full tablet report, sending...
I20260926 21:30:05.173350 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.156256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43779 (local address 127.0.97.62:36205)
0926 21:30:05.156420 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:05.156427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:05.157610 (+ 1183us) server_negotiation.cc:407] Connection header received
0926 21:30:05.157744 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:05.157753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:05.157844 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:05.157954 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:05.159422 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.160270 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.163162 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.163937 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.164055 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.166867 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:05.166876 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:05.166883 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:05.166932 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:05.169940 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.170897 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.170902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.170906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.171030 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.171468 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.171479 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.171482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.171957 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:05.172114 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:05.172803 (+ 689us) server_negotiation.cc:299] Negotiation successful
0926 21:30:05.173032 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 21:30:05.174748 2127 ts_manager.cc:194] Registered new tserver with Master: 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:05.175639 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41577
I20260926 21:30:05.176162 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.156618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36205 (local address 127.0.97.2:43779)
0926 21:30:05.157478 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:05.157503 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:05.157635 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:05.158168 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:05.158172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:05.158831 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:05.159229 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.159242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.160428 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.160441 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.162938 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.162961 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.164403 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.164416 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.164673 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.165415 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:05.165441 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:05.166659 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:05.170063 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:05.170069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:05.170073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:05.170756 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:05.171150 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:05.171156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:05.171170 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:05.171312 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:05.172072 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:05.172078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:05.172475 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 21:30:05.172647 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 21:30:05.173041 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":576,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:30:05.214318 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36205
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:05.536312 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:05.536592 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:05.536767 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:05.544097 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:05.544212 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:05.544361 2477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 21:30:05.544443 2477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:30:05.544574 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:05.555315 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:05.556589 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:05.557899 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:05.570935 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:05.571027 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:05.570969 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:05.571770 2477 server_base.cc:1034] running on GCE node
I20260926 21:30:05.572371 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:05.573240 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:05.574504 2477 hybrid_clock.cc:648] HybridClock initialized: now 1790458205574443 us; error 65 us; skew 500 ppm
I20260926 21:30:05.574811 2477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:05 dist-test-slave-87bt krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:05.577790 2477 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:30:05.579217 2477 webserver.cc:460] Webserver started at http://127.0.97.3:33109/ using document root <none> and password file <none>
I20260926 21:30:05.579875 2477 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:05.580016 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:05.580322 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:05.583194 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d"
format_stamp: "Formatted at 2026-09-26 21:30:05 on dist-test-slave-87bt"
server_key: "c55efbd9f92060126d0bda0b73373372"
server_key_iv: "3048a5e589298820004f2c9971556263"
server_key_version: "encryptionkey@0"
I20260926 21:30:05.583956 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d"
format_stamp: "Formatted at 2026-09-26 21:30:05 on dist-test-slave-87bt"
server_key: "c55efbd9f92060126d0bda0b73373372"
server_key_iv: "3048a5e589298820004f2c9971556263"
server_key_version: "encryptionkey@0"
I20260926 21:30:05.588949 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 21:30:05.592317 2492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:05.593771 2477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:05.593930 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d"
format_stamp: "Formatted at 2026-09-26 21:30:05 on dist-test-slave-87bt"
server_key: "c55efbd9f92060126d0bda0b73373372"
server_key_iv: "3048a5e589298820004f2c9971556263"
server_key_version: "encryptionkey@0"
I20260926 21:30:05.594113 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:05.616818 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:05.621523 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:05.621780 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:05.624502 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:05.624660 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:05.624770 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:05.624850 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:30:05 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:05.651058 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.625252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48267 (local address 127.0.97.62:36205)
0926 21:30:05.625455 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:05.625459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:05.627238 (+ 1779us) server_negotiation.cc:407] Connection header received
0926 21:30:05.629921 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:05.629930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:05.630050 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:05.630185 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:05.632812 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.633907 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.636377 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.636979 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.637133 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.642603 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:05.642611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:05.642619 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:05.642669 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:05.646688 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.648032 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.648038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.648041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.648229 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.648851 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.648857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.648861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.649315 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:05.649480 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:05.650445 (+ 965us) server_negotiation.cc:299] Negotiation successful
0926 21:30:05.650691 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:30:05.652622 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.626256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36205 (local address 127.0.97.3:48267)
0926 21:30:05.627083 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:05.627127 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:05.629469 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:05.630526 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:05.630542 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:05.631527 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:05.632497 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.632533 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.634084 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.634096 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.636150 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.636172 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.637240 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.637248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.637606 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.638608 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:05.638648 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:05.642257 (+ 3609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:05.646906 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:05.646919 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:05.646951 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:05.647801 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:05.648386 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:05.648395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:05.648401 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:05.648693 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:05.649461 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:05.649469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:05.649966 (+ 497us) client_negotiation.cc:769] Sending connection context
0926 21:30:05.650315 (+ 349us) client_negotiation.cc:240] Negotiation successful
0926 21:30:05.650947 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":365,"threads_started":1}
I20260926 21:30:05.673516 2477 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40663
I20260926 21:30:05.673550 2611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40663 every 8 connection(s)
I20260926 21:30:05.674332 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.659753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36205 (local address 127.0.97.3:51995)
0926 21:30:05.660634 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:05.660660 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:05.660825 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:05.661262 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:05.661266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:05.661951 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:05.662970 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.662982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.664170 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.664176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.665207 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.665220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.665970 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.665976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.666151 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.666828 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:05.666846 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:05.667795 (+ 949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:05.671668 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:05.671677 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:05.671681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:05.672159 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:05.672673 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:05.672678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:05.672681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:05.672840 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:05.673348 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:05.673353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:05.673559 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 21:30:05.673711 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 21:30:05.674023 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":553,"threads_started":1}
I20260926 21:30:05.674991 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:05.675163 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.660250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51995 (local address 127.0.97.62:36205)
0926 21:30:05.660412 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:05.660417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:05.660737 (+ 320us) server_negotiation.cc:407] Connection header received
0926 21:30:05.661017 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:05.661022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:05.661118 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:05.661238 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:05.663133 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.664024 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.665359 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.665779 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.665931 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.668006 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:05.668011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:05.668014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:05.668047 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:05.671416 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.672357 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.672363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.672369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.672534 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.672963 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.672966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.672968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.673227 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:05.673382 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:05.674701 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0926 21:30:05.674885 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 21:30:05.676249 2333 heartbeater.cc:503] Master 127.0.97.62:36205 was elected leader, sending a full tablet report...
I20260926 21:30:05.677449 2612 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36205
I20260926 21:30:05.678051 2612 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:05.678889 2612 heartbeater.cc:511] Master 127.0.97.62:36205 requested a full tablet report, sending...
I20260926 21:30:05.680121 2127 ts_manager.cc:194] Registered new tserver with Master: fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3:40663)
I20260926 21:30:05.680853 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48267
I20260926 21:30:05.682265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 2477
I20260926 21:30:05.682471 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:05.682927 388 external_mini_cluster.cc:1428] Setting key ef74d1f3d30a4a384721f021591d1958
I20260926 21:30:05.685791 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:30:05 dist-test-slave-87bt krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:30:05 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:05.732470 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.718673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.0.97.62:36205)
0926 21:30:05.718893 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:05.718900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:05.719079 (+ 179us) server_negotiation.cc:407] Connection header received
0926 21:30:05.719287 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:05.719293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:05.719362 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:05.719469 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:05.720859 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.721605 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.723000 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.723505 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.723673 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.727504 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:05.727513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:05.727522 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:05.727562 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:05.729873 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.730541 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.730547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.730550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.730676 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:05.731045 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:05.731051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:05.731055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:05.731331 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:05.731594 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:05.731879 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 21:30:05.732116 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:30:05.736222 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:05.742926 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:05.763983 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.754448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.0.97.1:38461)
0926 21:30:05.755107 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:05.755114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:05.755206 (+ 92us) server_negotiation.cc:407] Connection header received
0926 21:30:05.755545 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:05.755555 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:05.755901 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:05.756108 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:05.756877 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.758048 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.761133 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.762710 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.762891 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.763151 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:05.763493 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 21:30:05.763674 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":383,"threads_started":1}
I20260926 21:30:05.764091 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.754252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38461 (local address 127.0.0.1:43168)
0926 21:30:05.755066 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:05.755108 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:05.755302 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:05.756301 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:05.756308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:05.756346 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:05.756659 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.756673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.758184 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.758193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.760961 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.760975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.762974 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.762983 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.763217 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.763290 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 21:30:05.763426 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 21:30:05.763635 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":69,"thread_start_us":457,"threads_started":1}
I20260926 21:30:05.767405 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.756722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33179 (local address 127.0.0.1:41484)
0926 21:30:05.758672 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:05.758695 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:05.758805 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:05.759996 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:05.760001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:05.760026 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:05.760624 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.760639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.762178 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.762183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.764824 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.764840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.766514 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.766521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.766688 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.766723 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:30:05.766850 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 21:30:05.766998 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"mutex_wait_us":24,"thread_start_us":1635,"threads_started":1}
I20260926 21:30:05.767765 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.755458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40663 (local address 127.0.0.1:35380)
0926 21:30:05.756001 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:05.756021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:05.756143 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:05.760026 (+ 3883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:05.760031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:05.760055 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:05.760786 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.760802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.762916 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.762921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.765124 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:05.765137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.766928 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.766934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:05.767116 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.767149 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:30:05.767236 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 21:30:05.767373 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":59,"thread_start_us":275,"threads_started":1}
I20260926 21:30:05.767782 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.755875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.0.97.3:40663)
0926 21:30:05.759395 (+ 3520us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:05.759404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:05.759438 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:30:05.759534 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:05.759545 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:05.759793 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:05.760390 (+ 597us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:05.760998 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.762708 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.765321 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.766723 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.766852 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.767063 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:05.767292 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 21:30:05.767455 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3350,"thread_start_us":3168,"threads_started":1}
I20260926 21:30:05.768597 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:05.756349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.0.97.2:33179)
0926 21:30:05.756938 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:05.756949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:05.759557 (+ 2608us) server_negotiation.cc:407] Connection header received
0926 21:30:05.759654 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:05.759659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:05.759819 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:05.760029 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:05.760837 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.762005 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.765004 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:05.766293 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:05.766466 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:05.766668 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:05.768281 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0926 21:30:05.768399 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":502,"threads_started":1}
I20260926 21:30:05.768476 2264 tablet_service.cc:1467] Processing CreateTablet for tablet edfc32a81dab48a08f20d6392f03ce5d (DEFAULT_TABLE table=test-table [id=381a0fb365f645498fa395b08d665d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:05.769991 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edfc32a81dab48a08f20d6392f03ce5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:05.771152 2542 tablet_service.cc:1467] Processing CreateTablet for tablet edfc32a81dab48a08f20d6392f03ce5d (DEFAULT_TABLE table=test-table [id=381a0fb365f645498fa395b08d665d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:05.772245 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edfc32a81dab48a08f20d6392f03ce5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:05.771982 2403 tablet_service.cc:1467] Processing CreateTablet for tablet edfc32a81dab48a08f20d6392f03ce5d (DEFAULT_TABLE table=test-table [id=381a0fb365f645498fa395b08d665d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:05.773214 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edfc32a81dab48a08f20d6392f03ce5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:05.784421 2632 tablet_bootstrap.cc:492] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Bootstrap starting.
I20260926 21:30:05.786247 2631 tablet_bootstrap.cc:492] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Bootstrap starting.
I20260926 21:30:05.788599 2632 tablet_bootstrap.cc:654] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:05.789588 2632 log.cc:824] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:05.790349 2631 tablet_bootstrap.cc:654] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:05.791575 2631 log.cc:824] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:05.792405 2632 tablet_bootstrap.cc:492] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: No bootstrap required, opened a new log
I20260926 21:30:05.792655 2632 ts_tablet_manager.cc:1397] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 21:30:05.794114 2631 tablet_bootstrap.cc:492] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: No bootstrap required, opened a new log
I20260926 21:30:05.794358 2631 ts_tablet_manager.cc:1397] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:30:05.790818 2633 tablet_bootstrap.cc:492] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Bootstrap starting.
I20260926 21:30:05.798085 2633 tablet_bootstrap.cc:654] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:05.799407 2633 log.cc:824] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:05.801017 2632 raft_consensus.cc:348] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:05.801424 2632 raft_consensus.cc:374] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:05.801522 2632 raft_consensus.cc:729] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d92fcf19cf1445985a9f25f7bb826d6, State: Initialized, Role: FOLLOWER
I20260926 21:30:05.802141 2632 consensus_queue.cc:260] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:05.802590 2633 tablet_bootstrap.cc:492] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: No bootstrap required, opened a new log
I20260926 21:30:05.802862 2633 ts_tablet_manager.cc:1397] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 21:30:05.803870 2632 ts_tablet_manager.cc:1428] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 21:30:05.804857 2631 raft_consensus.cc:348] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:05.805285 2631 raft_consensus.cc:374] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:05.805399 2631 raft_consensus.cc:729] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Initialized, Role: FOLLOWER
I20260926 21:30:05.805985 2631 consensus_queue.cc:260] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:05.807062 2612 heartbeater.cc:503] Master 127.0.97.62:36205 was elected leader, sending a full tablet report...
I20260926 21:30:05.807643 2631 ts_tablet_manager.cc:1428] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260926 21:30:05.813970 2633 raft_consensus.cc:348] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:05.814316 2633 raft_consensus.cc:374] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:05.814389 2633 raft_consensus.cc:729] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bdd83ce79848918c37c032ecd8fffa, State: Initialized, Role: FOLLOWER
I20260926 21:30:05.814961 2633 consensus_queue.cc:260] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:05.815935 2473 heartbeater.cc:503] Master 127.0.97.62:36205 was elected leader, sending a full tablet report...
I20260926 21:30:05.816395 2633 ts_tablet_manager.cc:1428] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
W20260926 21:30:05.918157 2335 tablet.cc:2369] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:30:05.921645 2474 tablet.cc:2369] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:30:05.926759 2613 tablet.cc:2369] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:06.103967 2638 raft_consensus.cc:482] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:06.104331 2638 raft_consensus.cc:504] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:06.106081 2638 leader_election.cc:288] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:06.112704 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.106660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55351 (local address 127.0.97.1:38461)
0926 21:30:06.106872 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:06.106878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:06.106902 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:30:06.106988 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:06.106994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:06.107063 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:06.107195 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:06.107820 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.108761 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.110798 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.111796 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.111900 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.111975 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:06.112335 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 21:30:06.112446 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:30:06.112728 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.106536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38461 (local address 127.0.97.3:55351)
0926 21:30:06.106702 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:06.106722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:06.106864 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:06.107400 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:06.107404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:06.107428 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:06.107632 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.107644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.108914 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.108921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.110638 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.110651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.111965 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.111973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.112162 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.112199 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 21:30:06.112309 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 21:30:06.112451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 21:30:06.113336 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.107086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36929 (local address 127.0.97.2:33179)
0926 21:30:06.107294 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:06.107300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:06.107635 (+ 335us) server_negotiation.cc:407] Connection header received
0926 21:30:06.107772 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:06.107778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:06.107873 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:06.107952 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:06.108496 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.109494 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.111414 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.112363 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.112479 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.112574 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:06.113002 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 21:30:06.113167 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:30:06.113868 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.106934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33179 (local address 127.0.97.3:36929)
0926 21:30:06.107532 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:06.107549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:06.107648 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:06.107984 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:06.107988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:06.108011 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:06.108341 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.108351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.109634 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.109638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.111246 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.111259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.112589 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.112596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.112779 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.112814 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:30:06.113524 (+ 710us) client_negotiation.cc:240] Negotiation successful
0926 21:30:06.113625 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":3,"thread_start_us":347,"threads_started":1}
I20260926 21:30:06.113855 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfc32a81dab48a08f20d6392f03ce5d" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" is_pre_election: true
I20260926 21:30:06.114393 2284 raft_consensus.cc:2454] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 0.
I20260926 21:30:06.114866 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfc32a81dab48a08f20d6392f03ce5d" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bdd83ce79848918c37c032ecd8fffa" is_pre_election: true
I20260926 21:30:06.115123 2496 leader_election.cc:302] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d92fcf19cf1445985a9f25f7bb826d6, fb6b84a3e84c4df5a7cfdb775dc94d1d; no voters:
I20260926 21:30:06.115412 2423 raft_consensus.cc:2454] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 0.
I20260926 21:30:06.115540 2638 raft_consensus.cc:2781] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:06.115693 2638 raft_consensus.cc:482] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:06.115809 2638 raft_consensus.cc:3037] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.118067 2638 raft_consensus.cc:504] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:06.118654 2638 leader_election.cc:288] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 election: Requested vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:06.119123 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfc32a81dab48a08f20d6392f03ce5d" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6"
I20260926 21:30:06.119168 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfc32a81dab48a08f20d6392f03ce5d" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bdd83ce79848918c37c032ecd8fffa"
I20260926 21:30:06.119320 2284 raft_consensus.cc:3037] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.119354 2423 raft_consensus.cc:3037] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.121481 2423 raft_consensus.cc:2454] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 1.
I20260926 21:30:06.121495 2284 raft_consensus.cc:2454] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 1.
I20260926 21:30:06.121870 2496 leader_election.cc:302] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d92fcf19cf1445985a9f25f7bb826d6, fb6b84a3e84c4df5a7cfdb775dc94d1d; no voters:
I20260926 21:30:06.122181 2638 raft_consensus.cc:2781] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:06.122794 2638 raft_consensus.cc:686] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 LEADER]: Becoming Leader. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Running, Role: LEADER
I20260926 21:30:06.123271 2638 consensus_queue.cc:237] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } }
I20260926 21:30:06.128037 2126 catalog_manager.cc:5495] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } health_report { overall_health: UNKNOWN } } }
Sep 26 21:30:06 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:06.249548 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.236541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36205 (local address 127.0.0.1:34150)
0926 21:30:06.237519 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:06.237543 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:06.237666 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:06.237986 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:06.237989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:06.238741 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:06.238972 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.238981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.240064 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.240067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.241029 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.241041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.241722 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.241726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.241850 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.242565 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:06.242582 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:06.244676 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:06.247453 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:06.247462 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:06.247469 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:06.247869 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:06.248193 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:06.248196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:06.248199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:06.248323 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:06.248804 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:06.248810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:06.248953 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 21:30:06.249083 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 21:30:06.249271 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":598,"threads_started":1}
I20260926 21:30:06.249770 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.236565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.0.97.62:36205)
0926 21:30:06.237593 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:06.237597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:06.237620 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:30:06.237775 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:06.237778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:06.237839 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:06.237932 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:06.239122 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.239954 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.241151 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.241566 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.241656 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.244862 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:06.244867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:06.244871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:06.244897 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:06.247332 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:06.247965 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:06.247968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:06.247971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:06.248074 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:06.248411 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:06.248415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:06.248416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:06.248688 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:06.248807 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:06.249158 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 21:30:06.249402 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":849,"thread_start_us":692,"threads_started":1}
I20260926 21:30:06.254855 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34150:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 21:30:06.255916 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:06.262562 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb1606fee0408c80e59b8d6314b73b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec8399990cf14ead8e7d60477a987c2c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:06.262615 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb1606fee0408c80e59b8d6314b73b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec8399990cf14ead8e7d60477a987c2c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:06.262615 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb1606fee0408c80e59b8d6314b73b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec8399990cf14ead8e7d60477a987c2c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:06.263523 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb1606fee0408c80e59b8d6314b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:06.263531 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb1606fee0408c80e59b8d6314b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:06.263531 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb1606fee0408c80e59b8d6314b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:06.269531 2632 tablet_bootstrap.cc:492] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: Bootstrap starting.
I20260926 21:30:06.270669 2633 tablet_bootstrap.cc:492] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: Bootstrap starting.
I20260926 21:30:06.270998 2632 tablet_bootstrap.cc:654] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:06.272360 2633 tablet_bootstrap.cc:654] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:06.273195 2632 tablet_bootstrap.cc:492] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: No bootstrap required, opened a new log
I20260926 21:30:06.273320 2632 ts_tablet_manager.cc:1397] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 21:30:06.273350 2631 tablet_bootstrap.cc:492] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: Bootstrap starting.
I20260926 21:30:06.274062 2632 raft_consensus.cc:348] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.274309 2632 raft_consensus.cc:374] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:06.274386 2632 raft_consensus.cc:729] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d92fcf19cf1445985a9f25f7bb826d6, State: Initialized, Role: FOLLOWER
I20260926 21:30:06.274634 2632 consensus_queue.cc:260] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.274943 2631 tablet_bootstrap.cc:654] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:06.275146 2637 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.275563 2637 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:06.276151 2632 ts_tablet_manager.cc:1428] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:06.276510 2633 tablet_bootstrap.cc:492] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: No bootstrap required, opened a new log
I20260926 21:30:06.276672 2633 ts_tablet_manager.cc:1397] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260926 21:30:06.277498 2631 tablet_bootstrap.cc:492] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: No bootstrap required, opened a new log
I20260926 21:30:06.277459 2633 raft_consensus.cc:348] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.277763 2631 ts_tablet_manager.cc:1397] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260926 21:30:06.278273 2633 raft_consensus.cc:374] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:06.278396 2633 raft_consensus.cc:729] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bdd83ce79848918c37c032ecd8fffa, State: Initialized, Role: FOLLOWER
I20260926 21:30:06.278566 2631 raft_consensus.cc:348] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.278808 2631 raft_consensus.cc:374] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:06.278685 2633 consensus_queue.cc:260] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.278900 2631 raft_consensus.cc:729] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Initialized, Role: FOLLOWER
I20260926 21:30:06.279141 2631 consensus_queue.cc:260] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.279302 2633 ts_tablet_manager.cc:1428] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 21:30:06.279173 2639 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.279585 2639 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:06.279543 2638 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.279794 2638 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:06.280634 2631 ts_tablet_manager.cc:1428] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:06.287392 2638 raft_consensus.cc:482] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:06.287616 2638 raft_consensus.cc:504] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.288247 2638 leader_election.cc:288] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:06.288774 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb1606fee0408c80e59b8d6314b73b" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bdd83ce79848918c37c032ecd8fffa" is_pre_election: true
I20260926 21:30:06.288785 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb1606fee0408c80e59b8d6314b73b" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" is_pre_election: true
I20260926 21:30:06.289050 2423 raft_consensus.cc:2454] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 0.
I20260926 21:30:06.289068 2284 raft_consensus.cc:2454] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 0.
I20260926 21:30:06.289443 2496 leader_election.cc:302] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bdd83ce79848918c37c032ecd8fffa, fb6b84a3e84c4df5a7cfdb775dc94d1d; no voters:
I20260926 21:30:06.289716 2638 raft_consensus.cc:2781] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:06.289815 2638 raft_consensus.cc:482] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:06.289922 2638 raft_consensus.cc:3037] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.291940 2638 raft_consensus.cc:504] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.292454 2638 leader_election.cc:288] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 election: Requested vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:06.292845 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb1606fee0408c80e59b8d6314b73b" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6"
I20260926 21:30:06.292908 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb1606fee0408c80e59b8d6314b73b" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bdd83ce79848918c37c032ecd8fffa"
I20260926 21:30:06.293080 2284 raft_consensus.cc:3037] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.293098 2423 raft_consensus.cc:3037] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.296039 2284 raft_consensus.cc:2454] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 1.
I20260926 21:30:06.296032 2423 raft_consensus.cc:2454] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 1.
I20260926 21:30:06.296466 2496 leader_election.cc:302] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d92fcf19cf1445985a9f25f7bb826d6, fb6b84a3e84c4df5a7cfdb775dc94d1d; no voters:
I20260926 21:30:06.296737 2638 raft_consensus.cc:2781] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:06.296901 2638 raft_consensus.cc:686] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 LEADER]: Becoming Leader. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Running, Role: LEADER
I20260926 21:30:06.297134 2638 consensus_queue.cc:237] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.298682 2641 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: TxnStatusTablet state changed. Reason: New leader fb6b84a3e84c4df5a7cfdb775dc94d1d. Latest consensus state: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.298983 2641 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:06.299444 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 21:30:06.299773 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 21:30:06.300487 2127 catalog_manager.cc:5495] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } health_report { overall_health: HEALTHY } } }
I20260926 21:30:06.319219 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.312510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.0.97.3:40663)
0926 21:30:06.313064 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:06.313068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:06.313090 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:30:06.313169 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:06.313172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:06.313240 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:06.313365 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:06.313798 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.314696 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.316669 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.317318 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.317416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.317961 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:06.318606 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:06.318717 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:06.318850 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 21:30:06.318915 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":304,"threads_started":1}
I20260926 21:30:06.319290 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.312430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40663 (local address 127.0.0.1:35392)
0926 21:30:06.312670 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:06.312691 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:06.312823 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:06.313428 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:06.313433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:06.313448 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:06.313646 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.313658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.314832 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.314838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.316529 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.316545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.317471 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.317478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.317682 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.317758 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:06.318711 (+ 953us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:06.318732 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 21:30:06.318826 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 21:30:06.318987 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 21:30:06.327710 2284 raft_consensus.cc:1261] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Refusing update from remote peer fb6b84a3e84c4df5a7cfdb775dc94d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:06.327852 2423 raft_consensus.cc:1261] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Refusing update from remote peer fb6b84a3e84c4df5a7cfdb775dc94d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:06.328725 2641 consensus_queue.cc:1035] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:06.329137 2638 consensus_queue.cc:1035] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:06.337419 2637 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: TxnStatusTablet state changed. Reason: New leader fb6b84a3e84c4df5a7cfdb775dc94d1d. Latest consensus state: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.337706 2637 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:06.340010 2658 mvcc.cc:205] Tried to move back new op lower bound from 7333716813108301824 to 7333716812995796992. Current Snapshot: MvccSnapshot[applied={T|T < 7333716813108301824}]
I20260926 21:30:06.354265 2637 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.355435 2637 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P 3d92fcf19cf1445985a9f25f7bb826d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:06.356858 2639 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: TxnStatusTablet state changed. Reason: New leader fb6b84a3e84c4df5a7cfdb775dc94d1d. Latest consensus state: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.357149 2639 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:06.357761 2638 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.358050 2638 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:06.362577 2638 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.362916 2638 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P fb6b84a3e84c4df5a7cfdb775dc94d1d: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:06.366782 2639 tablet_replica.cc:439] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } } }
I20260926 21:30:06.367035 2639 tablet_replica.cc:442] T 5ffb1606fee0408c80e59b8d6314b73b P 19bdd83ce79848918c37c032ecd8fffa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:06.374136 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.367367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.0.97.3:40663)
0926 21:30:06.367543 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:06.367550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:06.368929 (+ 1379us) server_negotiation.cc:407] Connection header received
0926 21:30:06.369140 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:06.369147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:06.369222 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:06.369316 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:06.369774 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.370912 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.372247 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.372903 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.373041 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.373402 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:06.373530 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:06.373623 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:06.373719 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 21:30:06.373804 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:30:06.392546 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.386022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50383 (local address 127.0.97.3:40663)
0926 21:30:06.386214 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:06.386221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:06.386942 (+ 721us) server_negotiation.cc:407] Connection header received
0926 21:30:06.387113 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:06.387119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:06.387236 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:06.387342 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:06.387831 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.389014 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.390514 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.391188 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.391316 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.391770 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:06.391916 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:06.392008 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:06.392142 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 21:30:06.392218 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:30:06.392585 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:06.386054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40663 (local address 127.0.97.3:50383)
0926 21:30:06.386825 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:06.386845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:06.386990 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:06.387365 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:06.387368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:06.387378 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:06.387673 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.387684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.389171 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.389175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.390339 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:06.390350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:06.391384 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:06.391389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:06.391512 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:06.391568 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:06.392019 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:06.392027 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 21:30:06.392095 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 21:30:06.392196 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":423,"threads_started":1}
I20260926 21:30:06.403182 2423 raft_consensus.cc:1261] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Refusing update from remote peer fb6b84a3e84c4df5a7cfdb775dc94d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:06.403218 2284 raft_consensus.cc:1261] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Refusing update from remote peer fb6b84a3e84c4df5a7cfdb775dc94d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:06.403828 2641 consensus_queue.cc:1035] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:06.404137 2638 consensus_queue.cc:1035] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:06.410727 2659 mvcc.cc:205] Tried to move back new op lower bound from 7333716813421899776 to 7333716812284997632. Current Snapshot: MvccSnapshot[applied={T|T < 7333716813421899776}]
I20260926 21:30:06.472915 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:06.473217 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:06.476754 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: Sending DeleteTablet for 3 replicas of tablet edfc32a81dab48a08f20d6392f03ce5d
I20260926 21:30:06.477819 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet edfc32a81dab48a08f20d6392f03ce5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43168
I20260926 21:30:06.478041 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet edfc32a81dab48a08f20d6392f03ce5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41484
I20260926 21:30:06.479063 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet edfc32a81dab48a08f20d6392f03ce5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35380
I20260926 21:30:06.480242 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:06.481357 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:06.491137 2676 tablet_replica.cc:330] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: stopping tablet replica
I20260926 21:30:06.491372 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 00000399aab444c88e3c97e4e642c603 (DEFAULT_TABLE table=test-table [id=5f3e13e6724c43dc9e68335081f1facf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:06.491667 2676 raft_consensus.cc:2229] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:06.491987 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000399aab444c88e3c97e4e642c603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:06.492116 2676 raft_consensus.cc:2258] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:06.492385 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 00000399aab444c88e3c97e4e642c603 (DEFAULT_TABLE table=test-table [id=5f3e13e6724c43dc9e68335081f1facf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:06.492532 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 00000399aab444c88e3c97e4e642c603 (DEFAULT_TABLE table=test-table [id=5f3e13e6724c43dc9e68335081f1facf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:06.492928 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000399aab444c88e3c97e4e642c603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:06.492961 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000399aab444c88e3c97e4e642c603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:06.494887 2677 tablet_replica.cc:330] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: stopping tablet replica
I20260926 21:30:06.495420 2677 raft_consensus.cc:2229] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:06.496393 2677 raft_consensus.cc:2258] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:06.498410 2677 ts_tablet_manager.cc:1905] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:06.498504 2632 tablet_bootstrap.cc:492] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: Bootstrap starting.
I20260926 21:30:06.499857 2633 tablet_bootstrap.cc:492] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: Bootstrap starting.
I20260926 21:30:06.500126 2632 tablet_bootstrap.cc:654] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:06.501441 2633 tablet_bootstrap.cc:654] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:06.502409 2631 tablet_bootstrap.cc:492] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Bootstrap starting.
I20260926 21:30:06.504135 2631 tablet_bootstrap.cc:654] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:06.508582 2675 tablet_replica.cc:330] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: stopping tablet replica
I20260926 21:30:06.510635 2633 tablet_bootstrap.cc:492] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: No bootstrap required, opened a new log
I20260926 21:30:06.510898 2633 ts_tablet_manager.cc:1397] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260926 21:30:06.512310 2633 raft_consensus.cc:348] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.513065 2633 raft_consensus.cc:374] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:06.513240 2633 raft_consensus.cc:729] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bdd83ce79848918c37c032ecd8fffa, State: Initialized, Role: FOLLOWER
I20260926 21:30:06.513630 2675 raft_consensus.cc:2229] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:06.513670 2633 consensus_queue.cc:260] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.514107 2675 raft_consensus.cc:2258] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:06.515158 2633 ts_tablet_manager.cc:1428] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 21:30:06.516757 2675 ts_tablet_manager.cc:1905] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:06.517612 2676 ts_tablet_manager.cc:1905] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:06.525279 2631 tablet_bootstrap.cc:492] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: No bootstrap required, opened a new log
I20260926 21:30:06.525489 2631 ts_tablet_manager.cc:1397] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260926 21:30:06.525502 2676 ts_tablet_manager.cc:1918] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:06.525658 2676 log.cc:1196] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/edfc32a81dab48a08f20d6392f03ce5d
I20260926 21:30:06.525785 2675 ts_tablet_manager.cc:1918] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:06.525930 2675 log.cc:1196] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/edfc32a81dab48a08f20d6392f03ce5d
I20260926 21:30:06.526227 2676 ts_tablet_manager.cc:1939] T edfc32a81dab48a08f20d6392f03ce5d P 19bdd83ce79848918c37c032ecd8fffa: Deleting consensus metadata
I20260926 21:30:06.526400 2631 raft_consensus.cc:348] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.526580 2675 ts_tablet_manager.cc:1939] T edfc32a81dab48a08f20d6392f03ce5d P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting consensus metadata
I20260926 21:30:06.526695 2631 raft_consensus.cc:374] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:06.526775 2631 raft_consensus.cc:729] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Initialized, Role: FOLLOWER
I20260926 21:30:06.527014 2631 consensus_queue.cc:260] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.527616 2631 ts_tablet_manager.cc:1428] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 21:30:06.528831 2114 catalog_manager.cc:4841] TS 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461): tablet edfc32a81dab48a08f20d6392f03ce5d (table test-table [id=381a0fb365f645498fa395b08d665d0e]) successfully deleted
I20260926 21:30:06.529627 2114 catalog_manager.cc:4841] TS 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179): tablet edfc32a81dab48a08f20d6392f03ce5d (table test-table [id=381a0fb365f645498fa395b08d665d0e]) successfully deleted
I20260926 21:30:06.531944 2632 tablet_bootstrap.cc:492] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: No bootstrap required, opened a new log
I20260926 21:30:06.532111 2632 ts_tablet_manager.cc:1397] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.012s
I20260926 21:30:06.532967 2677 ts_tablet_manager.cc:1918] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:06.532877 2632 raft_consensus.cc:348] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.533105 2632 raft_consensus.cc:374] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:06.533159 2677 log.cc:1196] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/edfc32a81dab48a08f20d6392f03ce5d
I20260926 21:30:06.533182 2632 raft_consensus.cc:729] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d92fcf19cf1445985a9f25f7bb826d6, State: Initialized, Role: FOLLOWER
I20260926 21:30:06.533411 2632 consensus_queue.cc:260] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.533811 2677 ts_tablet_manager.cc:1939] T edfc32a81dab48a08f20d6392f03ce5d P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting consensus metadata
I20260926 21:30:06.534080 2632 ts_tablet_manager.cc:1428] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 21:30:06.535441 2114 catalog_manager.cc:4841] TS fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3:40663): tablet edfc32a81dab48a08f20d6392f03ce5d (table test-table [id=381a0fb365f645498fa395b08d665d0e]) successfully deleted
I20260926 21:30:06.609061 2638 raft_consensus.cc:482] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:06.609267 2638 raft_consensus.cc:504] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.610054 2638 leader_election.cc:288] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:06.610587 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000399aab444c88e3c97e4e642c603" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" is_pre_election: true
I20260926 21:30:06.610662 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000399aab444c88e3c97e4e642c603" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bdd83ce79848918c37c032ecd8fffa" is_pre_election: true
I20260926 21:30:06.610906 2283 raft_consensus.cc:2454] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 0.
I20260926 21:30:06.610976 2422 raft_consensus.cc:2454] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 0.
I20260926 21:30:06.611466 2496 leader_election.cc:302] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bdd83ce79848918c37c032ecd8fffa, fb6b84a3e84c4df5a7cfdb775dc94d1d; no voters:
I20260926 21:30:06.611786 2638 raft_consensus.cc:2781] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:06.611915 2638 raft_consensus.cc:482] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:06.612012 2638 raft_consensus.cc:3037] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.614567 2638 raft_consensus.cc:504] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.615185 2638 leader_election.cc:288] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 election: Requested vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:06.615770 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000399aab444c88e3c97e4e642c603" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6"
I20260926 21:30:06.615803 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000399aab444c88e3c97e4e642c603" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bdd83ce79848918c37c032ecd8fffa"
I20260926 21:30:06.615998 2283 raft_consensus.cc:3037] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.616062 2422 raft_consensus.cc:3037] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:06.618183 2283 raft_consensus.cc:2454] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 1.
I20260926 21:30:06.618660 2496 leader_election.cc:302] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d92fcf19cf1445985a9f25f7bb826d6, fb6b84a3e84c4df5a7cfdb775dc94d1d; no voters:
I20260926 21:30:06.618973 2638 raft_consensus.cc:2781] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:06.619136 2638 raft_consensus.cc:686] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 LEADER]: Becoming Leader. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Running, Role: LEADER
I20260926 21:30:06.619141 2422 raft_consensus.cc:2454] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 1.
I20260926 21:30:06.619477 2638 consensus_queue.cc:237] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:06.622246 2125 catalog_manager.cc:5495] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } health_report { overall_health: HEALTHY } } }
I20260926 21:30:06.651381 2423 raft_consensus.cc:1261] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Refusing update from remote peer fb6b84a3e84c4df5a7cfdb775dc94d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:06.652137 2638 consensus_queue.cc:1035] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:06.652462 2283 raft_consensus.cc:1261] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Refusing update from remote peer fb6b84a3e84c4df5a7cfdb775dc94d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:06.653115 2638 consensus_queue.cc:1035] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:06.660812 2660 mvcc.cc:205] Tried to move back new op lower bound from 7333716814438588416 to 7333716814315438080. Current Snapshot: MvccSnapshot[applied={T|T < 7333716814438588416}]
W20260926 21:30:06.691725 2658 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 21:30:08.289405 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:08.289605 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:08.293776 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: Sending DeleteTablet for 3 replicas of tablet 00000399aab444c88e3c97e4e642c603
I20260926 21:30:08.294600 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 00000399aab444c88e3c97e4e642c603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43168
I20260926 21:30:08.294638 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 00000399aab444c88e3c97e4e642c603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41484
I20260926 21:30:08.294806 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 00000399aab444c88e3c97e4e642c603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35380
I20260926 21:30:08.310295 2705 tablet_replica.cc:330] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: stopping tablet replica
I20260926 21:30:08.310719 2706 tablet_replica.cc:330] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: stopping tablet replica
I20260926 21:30:08.310911 2704 tablet_replica.cc:330] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: stopping tablet replica
I20260926 21:30:08.310981 2705 raft_consensus.cc:2229] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:08.311038 2706 raft_consensus.cc:2229] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:08.311153 2704 raft_consensus.cc:2229] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:08.311368 2705 raft_consensus.cc:2258] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:08.311447 2704 raft_consensus.cc:2258] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:08.311896 2706 raft_consensus.cc:2258] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:08.312897 2705 ts_tablet_manager.cc:1905] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:08.312903 2704 ts_tablet_manager.cc:1905] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:08.312987 2706 ts_tablet_manager.cc:1905] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:08.320133 2704 ts_tablet_manager.cc:1918] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:30:08.320133 2706 ts_tablet_manager.cc:1918] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:30:08.320327 2704 log.cc:1196] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/00000399aab444c88e3c97e4e642c603
I20260926 21:30:08.320363 2706 log.cc:1196] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/00000399aab444c88e3c97e4e642c603
I20260926 21:30:08.320634 2705 ts_tablet_manager.cc:1918] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:30:08.320851 2704 ts_tablet_manager.cc:1939] T 00000399aab444c88e3c97e4e642c603 P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting consensus metadata
I20260926 21:30:08.320855 2705 log.cc:1196] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/00000399aab444c88e3c97e4e642c603
I20260926 21:30:08.320866 2706 ts_tablet_manager.cc:1939] T 00000399aab444c88e3c97e4e642c603 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting consensus metadata
I20260926 21:30:08.321403 2705 ts_tablet_manager.cc:1939] T 00000399aab444c88e3c97e4e642c603 P 19bdd83ce79848918c37c032ecd8fffa: Deleting consensus metadata
I20260926 21:30:08.321913 2114 catalog_manager.cc:4841] TS fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3:40663): tablet 00000399aab444c88e3c97e4e642c603 (table test-table [id=5f3e13e6724c43dc9e68335081f1facf]) successfully deleted
I20260926 21:30:08.322227 2114 catalog_manager.cc:4841] TS 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461): tablet 00000399aab444c88e3c97e4e642c603 (table test-table [id=5f3e13e6724c43dc9e68335081f1facf]) successfully deleted
I20260926 21:30:08.322549 2114 catalog_manager.cc:4841] TS 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179): tablet 00000399aab444c88e3c97e4e642c603 (table test-table [id=5f3e13e6724c43dc9e68335081f1facf]) successfully deleted
Sep 26 21:30:08 dist-test-slave-87bt krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:30:08 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:08.353524 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.339388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.0.97.62:36205)
0926 21:30:08.339991 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:08.339997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:08.340140 (+ 143us) server_negotiation.cc:407] Connection header received
0926 21:30:08.340381 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:08.340385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:08.340445 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:08.340558 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:08.341837 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.342703 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.343937 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.344355 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.344495 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.348493 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:08.348498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:08.348501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:08.348535 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:08.351124 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:08.351724 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:08.351731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:08.351734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:08.351887 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:08.352356 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:08.352360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:08.352362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:08.352630 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:08.352812 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:08.353058 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 21:30:08.353255 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":288,"threads_started":1}
I20260926 21:30:08.357368 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:08.358007 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:08.364569 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 03abee4990e2409e9b8dd418a5cb420c (DEFAULT_TABLE table=test-table [id=8fd1d243f99b415db5503f1156b05a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:08.364744 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 03abee4990e2409e9b8dd418a5cb420c (DEFAULT_TABLE table=test-table [id=8fd1d243f99b415db5503f1156b05a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:08.365053 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03abee4990e2409e9b8dd418a5cb420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:08.365135 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03abee4990e2409e9b8dd418a5cb420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:08.365128 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 03abee4990e2409e9b8dd418a5cb420c (DEFAULT_TABLE table=test-table [id=8fd1d243f99b415db5503f1156b05a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:08.365525 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03abee4990e2409e9b8dd418a5cb420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:08.370779 2715 tablet_bootstrap.cc:492] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: Bootstrap starting.
I20260926 21:30:08.371935 2717 tablet_bootstrap.cc:492] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: Bootstrap starting.
I20260926 21:30:08.372953 2716 tablet_bootstrap.cc:492] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: Bootstrap starting.
I20260926 21:30:08.373770 2715 tablet_bootstrap.cc:654] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:08.374584 2716 tablet_bootstrap.cc:654] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:08.376048 2715 tablet_bootstrap.cc:492] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: No bootstrap required, opened a new log
I20260926 21:30:08.376200 2715 ts_tablet_manager.cc:1397] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 21:30:08.376296 2717 tablet_bootstrap.cc:654] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:08.376821 2716 tablet_bootstrap.cc:492] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: No bootstrap required, opened a new log
I20260926 21:30:08.377002 2716 ts_tablet_manager.cc:1397] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260926 21:30:08.377070 2715 raft_consensus.cc:348] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.377382 2715 raft_consensus.cc:374] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:08.377513 2715 raft_consensus.cc:729] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Initialized, Role: FOLLOWER
I20260926 21:30:08.377832 2715 consensus_queue.cc:260] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.377820 2716 raft_consensus.cc:348] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.378057 2716 raft_consensus.cc:374] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:08.378640 2716 raft_consensus.cc:729] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d92fcf19cf1445985a9f25f7bb826d6, State: Initialized, Role: FOLLOWER
I20260926 21:30:08.379073 2715 ts_tablet_manager.cc:1428] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 21:30:08.378939 2716 consensus_queue.cc:260] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.379819 2716 ts_tablet_manager.cc:1428] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 21:30:08.381820 2717 tablet_bootstrap.cc:492] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: No bootstrap required, opened a new log
I20260926 21:30:08.381999 2717 ts_tablet_manager.cc:1397] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 21:30:08.382836 2717 raft_consensus.cc:348] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.383086 2717 raft_consensus.cc:374] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:08.383201 2717 raft_consensus.cc:729] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bdd83ce79848918c37c032ecd8fffa, State: Initialized, Role: FOLLOWER
I20260926 21:30:08.383432 2717 consensus_queue.cc:260] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.384101 2717 ts_tablet_manager.cc:1428] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:08.556797 2696 raft_consensus.cc:482] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:08.556972 2696 raft_consensus.cc:504] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.557529 2696 leader_election.cc:288] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:08.558197 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03abee4990e2409e9b8dd418a5cb420c" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bdd83ce79848918c37c032ecd8fffa" is_pre_election: true
I20260926 21:30:08.558250 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03abee4990e2409e9b8dd418a5cb420c" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" is_pre_election: true
I20260926 21:30:08.558517 2422 raft_consensus.cc:2454] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 0.
I20260926 21:30:08.558532 2284 raft_consensus.cc:2454] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 0.
I20260926 21:30:08.559026 2496 leader_election.cc:302] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bdd83ce79848918c37c032ecd8fffa, fb6b84a3e84c4df5a7cfdb775dc94d1d; no voters:
I20260926 21:30:08.559325 2696 raft_consensus.cc:2781] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:08.559451 2696 raft_consensus.cc:482] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:08.559553 2696 raft_consensus.cc:3037] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:08.561523 2696 raft_consensus.cc:504] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.562089 2696 leader_election.cc:288] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 election: Requested vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179)
I20260926 21:30:08.562721 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03abee4990e2409e9b8dd418a5cb420c" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bdd83ce79848918c37c032ecd8fffa"
I20260926 21:30:08.562741 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03abee4990e2409e9b8dd418a5cb420c" candidate_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6"
I20260926 21:30:08.562959 2284 raft_consensus.cc:3037] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:08.562981 2422 raft_consensus.cc:3037] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:08.566330 2422 raft_consensus.cc:2454] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 1.
I20260926 21:30:08.566344 2284 raft_consensus.cc:2454] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6b84a3e84c4df5a7cfdb775dc94d1d in term 1.
I20260926 21:30:08.566839 2496 leader_election.cc:302] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bdd83ce79848918c37c032ecd8fffa, fb6b84a3e84c4df5a7cfdb775dc94d1d; no voters:
I20260926 21:30:08.567128 2696 raft_consensus.cc:2781] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:08.567320 2696 raft_consensus.cc:686] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 LEADER]: Becoming Leader. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Running, Role: LEADER
I20260926 21:30:08.567615 2696 consensus_queue.cc:237] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.570297 2127 catalog_manager.cc:5495] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } health_report { overall_health: HEALTHY } } }
I20260926 21:30:08.619468 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.614592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.0.97.3:40663)
0926 21:30:08.615139 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:08.615144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:08.615165 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:30:08.615215 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:08.615218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:08.615287 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:08.615370 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:08.615757 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.616593 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.617901 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.618319 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.618408 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.618834 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:08.618926 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:08.618992 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:08.619147 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 21:30:08.619204 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":301,"threads_started":1}
I20260926 21:30:08.627592 2284 raft_consensus.cc:1261] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Refusing update from remote peer fb6b84a3e84c4df5a7cfdb775dc94d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:08.628091 2422 raft_consensus.cc:1261] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Refusing update from remote peer fb6b84a3e84c4df5a7cfdb775dc94d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:08.628278 2687 consensus_queue.cc:1035] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:08.628651 2665 consensus_queue.cc:1035] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:08.670215 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:08.670497 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:08.673580 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: Sending DeleteTablet for 3 replicas of tablet 03abee4990e2409e9b8dd418a5cb420c
I20260926 21:30:08.675166 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 03abee4990e2409e9b8dd418a5cb420c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43168
I20260926 21:30:08.675474 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 03abee4990e2409e9b8dd418a5cb420c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41484
I20260926 21:30:08.675487 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 21:30:08.675742 2704 tablet_replica.cc:330] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: stopping tablet replica
I20260926 21:30:08.675740 2705 tablet_replica.cc:330] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: stopping tablet replica
I20260926 21:30:08.675916 2704 raft_consensus.cc:2229] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:08.675925 2705 raft_consensus.cc:2229] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:08.676084 2705 raft_consensus.cc:2258] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 21:30:08.676083 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:08.676088 2704 raft_consensus.cc:2258] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:08.678158 2704 ts_tablet_manager.cc:1905] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:08.678304 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 03abee4990e2409e9b8dd418a5cb420c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35380
I20260926 21:30:08.678591 2706 tablet_replica.cc:330] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: stopping tablet replica
I20260926 21:30:08.678776 2706 raft_consensus.cc:2229] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:08.679090 2706 raft_consensus.cc:2258] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:08.682703 2706 ts_tablet_manager.cc:1905] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:08.682938 2705 ts_tablet_manager.cc:1905] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:08.683845 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 2880c739fa684225b785803af0280b82 (DEFAULT_TABLE table=test-table [id=6333466d3ae74a80b7f1e068b665f460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:08.684280 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880c739fa684225b785803af0280b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:08.684648 2704 ts_tablet_manager.cc:1918] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:08.684778 2704 log.cc:1196] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/03abee4990e2409e9b8dd418a5cb420c
I20260926 21:30:08.685159 2704 ts_tablet_manager.cc:1939] T 03abee4990e2409e9b8dd418a5cb420c P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting consensus metadata
I20260926 21:30:08.686164 2114 catalog_manager.cc:4841] TS 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461): tablet 03abee4990e2409e9b8dd418a5cb420c (table test-table [id=8fd1d243f99b415db5503f1156b05a49]) successfully deleted
I20260926 21:30:08.689260 2706 ts_tablet_manager.cc:1918] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:08.689471 2706 log.cc:1196] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/03abee4990e2409e9b8dd418a5cb420c
I20260926 21:30:08.689852 2705 ts_tablet_manager.cc:1918] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:08.689936 2706 ts_tablet_manager.cc:1939] T 03abee4990e2409e9b8dd418a5cb420c P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting consensus metadata
I20260926 21:30:08.690235 2705 log.cc:1196] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/03abee4990e2409e9b8dd418a5cb420c
I20260926 21:30:08.690687 2705 ts_tablet_manager.cc:1939] T 03abee4990e2409e9b8dd418a5cb420c P 19bdd83ce79848918c37c032ecd8fffa: Deleting consensus metadata
I20260926 21:30:08.690830 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 2880c739fa684225b785803af0280b82 (DEFAULT_TABLE table=test-table [id=6333466d3ae74a80b7f1e068b665f460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:08.691273 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880c739fa684225b785803af0280b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:08.694706 2114 catalog_manager.cc:4841] TS fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3:40663): tablet 03abee4990e2409e9b8dd418a5cb420c (table test-table [id=8fd1d243f99b415db5503f1156b05a49]) successfully deleted
I20260926 21:30:08.695195 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 2880c739fa684225b785803af0280b82 (DEFAULT_TABLE table=test-table [id=6333466d3ae74a80b7f1e068b665f460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:08.695650 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880c739fa684225b785803af0280b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:08.695936 2717 tablet_bootstrap.cc:492] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: Bootstrap starting.
I20260926 21:30:08.697602 2717 tablet_bootstrap.cc:654] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:08.698975 2114 catalog_manager.cc:4841] TS 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179): tablet 03abee4990e2409e9b8dd418a5cb420c (table test-table [id=8fd1d243f99b415db5503f1156b05a49]) successfully deleted
I20260926 21:30:08.701933 2716 tablet_bootstrap.cc:492] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: Bootstrap starting.
I20260926 21:30:08.702179 2715 tablet_bootstrap.cc:492] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Bootstrap starting.
I20260926 21:30:08.703586 2716 tablet_bootstrap.cc:654] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:08.704372 2715 tablet_bootstrap.cc:654] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:08.705502 2717 tablet_bootstrap.cc:492] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: No bootstrap required, opened a new log
I20260926 21:30:08.705679 2717 ts_tablet_manager.cc:1397] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260926 21:30:08.706450 2717 raft_consensus.cc:348] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.706795 2717 raft_consensus.cc:374] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:08.706898 2717 raft_consensus.cc:729] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bdd83ce79848918c37c032ecd8fffa, State: Initialized, Role: FOLLOWER
I20260926 21:30:08.707106 2715 tablet_bootstrap.cc:492] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: No bootstrap required, opened a new log
I20260926 21:30:08.707267 2715 ts_tablet_manager.cc:1397] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 21:30:08.707229 2717 consensus_queue.cc:260] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.707964 2715 raft_consensus.cc:348] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.708189 2715 raft_consensus.cc:374] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:08.708284 2715 raft_consensus.cc:729] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6b84a3e84c4df5a7cfdb775dc94d1d, State: Initialized, Role: FOLLOWER
I20260926 21:30:08.708503 2717 ts_tablet_manager.cc:1428] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 21:30:08.711159 2715 consensus_queue.cc:260] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.711850 2715 ts_tablet_manager.cc:1428] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260926 21:30:08.713989 2716 tablet_bootstrap.cc:492] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: No bootstrap required, opened a new log
I20260926 21:30:08.714178 2716 ts_tablet_manager.cc:1397] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20260926 21:30:08.715009 2716 raft_consensus.cc:348] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.715389 2716 raft_consensus.cc:374] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:08.715507 2716 raft_consensus.cc:729] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d92fcf19cf1445985a9f25f7bb826d6, State: Initialized, Role: FOLLOWER
I20260926 21:30:08.715773 2716 consensus_queue.cc:260] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.716492 2716 ts_tablet_manager.cc:1428] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:08.894253 2722 raft_consensus.cc:482] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:08.894573 2722 raft_consensus.cc:504] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.896198 2722 leader_election.cc:288] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3:40663)
I20260926 21:30:08.904399 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.897337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36541 (local address 127.0.97.3:40663)
0926 21:30:08.897542 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:08.897550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:08.898071 (+ 521us) server_negotiation.cc:407] Connection header received
0926 21:30:08.898283 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:08.898288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:08.898351 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:08.898527 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:08.899178 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.900051 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.902368 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.903333 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.903484 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.903603 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:08.904021 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 21:30:08.904160 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 21:30:08.904628 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.897195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40663 (local address 127.0.97.2:36541)
0926 21:30:08.897965 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:08.897983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:08.898120 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:08.898559 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:08.898564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:08.898603 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:08.899021 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:08.899030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.900215 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.900219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:08.902225 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:08.902241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.903640 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.903650 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:08.903840 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.903873 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:30:08.903994 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 21:30:08.904130 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":2,"thread_start_us":436,"threads_started":1}
I20260926 21:30:08.904664 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.896604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43519 (local address 127.0.97.1:38461)
0926 21:30:08.897976 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:08.897983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:08.898011 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:30:08.898088 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:08.898093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:08.898170 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:08.898263 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:08.898922 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.899841 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.902106 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.903632 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.903731 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.903821 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:08.904151 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 21:30:08.904327 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":1091,"threads_started":1}
I20260926 21:30:08.904635 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.896461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38461 (local address 127.0.97.2:43519)
0926 21:30:08.896980 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:08.896997 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:08.897283 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:08.898283 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:08.898287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:08.898309 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:08.898727 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:08.898742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.899972 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.899977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:08.901970 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:08.901982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.903825 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.903831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:08.904006 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.904034 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 21:30:08.904120 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 21:30:08.904221 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":6,"thread_start_us":250,"threads_started":1}
I20260926 21:30:08.905288 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880c739fa684225b785803af0280b82" candidate_uuid: "19bdd83ce79848918c37c032ecd8fffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" is_pre_election: true
I20260926 21:30:08.905531 2284 raft_consensus.cc:2454] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bdd83ce79848918c37c032ecd8fffa in term 0.
I20260926 21:30:08.905674 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880c739fa684225b785803af0280b82" candidate_uuid: "19bdd83ce79848918c37c032ecd8fffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" is_pre_election: true
I20260926 21:30:08.906021 2357 leader_election.cc:302] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bdd83ce79848918c37c032ecd8fffa, 3d92fcf19cf1445985a9f25f7bb826d6; no voters:
I20260926 21:30:08.906103 2562 raft_consensus.cc:2454] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bdd83ce79848918c37c032ecd8fffa in term 0.
I20260926 21:30:08.906461 2722 raft_consensus.cc:2781] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:08.906605 2722 raft_consensus.cc:482] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:08.906718 2722 raft_consensus.cc:3037] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:08.908855 2722 raft_consensus.cc:504] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.909410 2722 leader_election.cc:288] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [CANDIDATE]: Term 1 election: Requested vote from peers 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461), fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3:40663)
I20260926 21:30:08.909842 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880c739fa684225b785803af0280b82" candidate_uuid: "19bdd83ce79848918c37c032ecd8fffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d92fcf19cf1445985a9f25f7bb826d6"
I20260926 21:30:08.909893 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880c739fa684225b785803af0280b82" candidate_uuid: "19bdd83ce79848918c37c032ecd8fffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d"
I20260926 21:30:08.910018 2284 raft_consensus.cc:3037] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:08.910125 2562 raft_consensus.cc:3037] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:08.912081 2284 raft_consensus.cc:2454] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bdd83ce79848918c37c032ecd8fffa in term 1.
I20260926 21:30:08.912292 2562 raft_consensus.cc:2454] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bdd83ce79848918c37c032ecd8fffa in term 1.
I20260926 21:30:08.912508 2357 leader_election.cc:302] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bdd83ce79848918c37c032ecd8fffa, 3d92fcf19cf1445985a9f25f7bb826d6; no voters:
I20260926 21:30:08.912788 2722 raft_consensus.cc:2781] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:08.913213 2722 raft_consensus.cc:686] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 1 LEADER]: Becoming Leader. State: Replica: 19bdd83ce79848918c37c032ecd8fffa, State: Running, Role: LEADER
I20260926 21:30:08.913581 2722 consensus_queue.cc:237] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } }
I20260926 21:30:08.917512 2126 catalog_manager.cc:5495] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa reported cstate change: term changed from 0 to 1, leader changed from <none> to 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2). New cstate: current_term: 1 leader_uuid: "19bdd83ce79848918c37c032ecd8fffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bdd83ce79848918c37c032ecd8fffa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 } health_report { overall_health: UNKNOWN } } }
I20260926 21:30:08.964944 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.959102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.0.97.2:33179)
0926 21:30:08.959655 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:08.959659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:08.959677 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:08.959752 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:08.959756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:08.959824 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:08.959959 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:08.960303 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.961084 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.962414 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.962968 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.963082 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.963569 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:08.964199 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:08.964322 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:08.964474 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 21:30:08.964540 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":291,"threads_started":1}
I20260926 21:30:08.980108 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.974347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40111 (local address 127.0.97.3:40663)
0926 21:30:08.974538 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:08.974542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:08.975479 (+ 937us) server_negotiation.cc:407] Connection header received
0926 21:30:08.975687 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:08.975692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:08.975762 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:08.975851 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:08.976305 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.977053 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.978390 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.978854 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.979000 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.979417 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:08.979511 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:08.979602 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:08.979740 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 21:30:08.979818 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:30:08.980211 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:08.974360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40663 (local address 127.0.97.2:40111)
0926 21:30:08.975377 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:08.975392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:08.975552 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:08.975876 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:08.975881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:08.975893 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:08.976171 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:08.976179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.977188 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.977194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:08.978250 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:08.978260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:08.979027 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:08.979034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:08.979185 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:08.979237 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:08.979602 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:08.979615 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 21:30:08.979712 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 21:30:08.979841 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":709,"threads_started":1}
I20260926 21:30:08.988881 2284 raft_consensus.cc:1261] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Refusing update from remote peer 19bdd83ce79848918c37c032ecd8fffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:08.989006 2562 raft_consensus.cc:1261] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Refusing update from remote peer 19bdd83ce79848918c37c032ecd8fffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:08.989545 2736 consensus_queue.cc:1035] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d92fcf19cf1445985a9f25f7bb826d6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:08.989956 2722 consensus_queue.cc:1035] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6b84a3e84c4df5a7cfdb775dc94d1d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:09.024624 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:09.019003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33179 (local address 127.0.97.3:58911)
0926 21:30:09.019640 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:09.019657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:09.019777 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:09.020042 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:09.020045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:09.020054 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:09.020349 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:09.020356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:09.021299 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:09.021303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:09.022413 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:09.022454 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:09.023125 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:09.023130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:09.023280 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:09.023315 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:09.023848 (+ 533us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:09.023863 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 21:30:09.023974 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 21:30:09.024189 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":347,"threads_started":1}
I20260926 21:30:09.024652 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:09.019112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58911 (local address 127.0.97.2:33179)
0926 21:30:09.019308 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:09.019315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:09.019738 (+ 423us) server_negotiation.cc:407] Connection header received
0926 21:30:09.019870 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:09.019874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:09.019936 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:09.020009 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:09.020477 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:09.021170 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:09.022568 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:09.022969 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:09.023077 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:09.023595 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:09.023727 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:09.023819 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:09.024335 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 21:30:09.024417 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 21:30:09.025768 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 21:30:10.633188 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:10.633428 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:10.637166 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9361219265d24bed8c606793953f2182: Sending DeleteTablet for 3 replicas of tablet 2880c739fa684225b785803af0280b82
I20260926 21:30:10.637957 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880c739fa684225b785803af0280b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43168
I20260926 21:30:10.637986 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880c739fa684225b785803af0280b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41484
I20260926 21:30:10.638218 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880c739fa684225b785803af0280b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35380
I20260926 21:30:10.638782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2198
I20260926 21:30:10.650319 2772 tablet_replica.cc:330] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: stopping tablet replica
I20260926 21:30:10.650502 2771 tablet_replica.cc:330] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: stopping tablet replica
I20260926 21:30:10.650578 2773 tablet_replica.cc:330] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: stopping tablet replica
I20260926 21:30:10.650715 2772 raft_consensus.cc:2229] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:10.650787 2771 raft_consensus.cc:2229] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:10.650992 2773 raft_consensus.cc:2229] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:10.651021 2772 raft_consensus.cc:2258] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:10.651353 2773 raft_consensus.cc:2258] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:10.651417 2771 raft_consensus.cc:2258] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:10.652407 2771 ts_tablet_manager.cc:1905] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:10.652407 2772 ts_tablet_manager.cc:1905] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:10.653196 2773 ts_tablet_manager.cc:1905] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:10.658946 2771 ts_tablet_manager.cc:1918] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:30:10.659037 2772 ts_tablet_manager.cc:1918] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:30:10.659162 2771 log.cc:1196] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/2880c739fa684225b785803af0280b82
I20260926 21:30:10.659209 2772 log.cc:1196] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/2880c739fa684225b785803af0280b82
I20260926 21:30:10.659716 2772 ts_tablet_manager.cc:1939] T 2880c739fa684225b785803af0280b82 P 3d92fcf19cf1445985a9f25f7bb826d6: Deleting consensus metadata
I20260926 21:30:10.659858 2771 ts_tablet_manager.cc:1939] T 2880c739fa684225b785803af0280b82 P 19bdd83ce79848918c37c032ecd8fffa: Deleting consensus metadata
I20260926 21:30:10.660295 2773 ts_tablet_manager.cc:1918] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:30:10.660501 2773 log.cc:1196] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/2880c739fa684225b785803af0280b82
I20260926 21:30:10.660851 2114 catalog_manager.cc:4841] TS 19bdd83ce79848918c37c032ecd8fffa (127.0.97.2:33179): tablet 2880c739fa684225b785803af0280b82 (table test-table [id=6333466d3ae74a80b7f1e068b665f460]) successfully deleted
I20260926 21:30:10.661161 2114 catalog_manager.cc:4841] TS 3d92fcf19cf1445985a9f25f7bb826d6 (127.0.97.1:38461): tablet 2880c739fa684225b785803af0280b82 (table test-table [id=6333466d3ae74a80b7f1e068b665f460]) successfully deleted
I20260926 21:30:10.661638 2773 ts_tablet_manager.cc:1939] T 2880c739fa684225b785803af0280b82 P fb6b84a3e84c4df5a7cfdb775dc94d1d: Deleting consensus metadata
I20260926 21:30:10.662526 2114 catalog_manager.cc:4841] TS fb6b84a3e84c4df5a7cfdb775dc94d1d (127.0.97.3:40663): tablet 2880c739fa684225b785803af0280b82 (table test-table [id=6333466d3ae74a80b7f1e068b665f460]) successfully deleted
Sep 26 21:30:10 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:10.675700 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:10.642539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.0.97.1:38461)
0926 21:30:10.647447 (+ 4908us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:10.647456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:10.653427 (+ 5971us) server_negotiation.cc:407] Connection header received
0926 21:30:10.653896 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:10.653902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:10.653984 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:10.654070 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:10.655916 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:10.657144 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:10.659414 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:10.660076 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:10.660194 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:10.669878 (+ 9684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:10.669894 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:10.669909 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:10.669943 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:10.672815 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:10.673495 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:10.673501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:10.673504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:10.673645 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:10.674024 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:10.674034 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:10.674036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:10.674362 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:10.674757 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:10.674991 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 21:30:10.675218 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4735,"thread_start_us":4579,"threads_started":1}
W20260926 21:30:10.676362 2328 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58316
W20260926 21:30:10.676992 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2198: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:10.677220 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 2198
I20260926 21:30:10.695677 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2338
Sep 26 21:30:10 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:30:10.710803 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:10.696425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.0.97.2:33179)
0926 21:30:10.696952 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:10.696956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:10.696973 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:30:10.697022 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:10.697025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:10.697080 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:10.697179 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:10.698504 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:10.699346 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:10.700572 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:10.701040 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:10.701135 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:10.705312 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:10.705325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:10.705341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:10.705369 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:10.708182 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:10.708809 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:10.708817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:10.708819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:10.708937 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:10.709296 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:10.709302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:10.709306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:10.709632 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:10.709950 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:10.710214 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 21:30:10.710440 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":307,"threads_started":1}
W20260926 21:30:10.711438 2468 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45498
W20260926 21:30:10.712100 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2338: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:10.712251 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 2338
I20260926 21:30:10.735054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2477
Sep 26 21:30:10 dist-test-slave-87bt krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:30:10.749873 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:10.735965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.0.97.3:40663)
0926 21:30:10.736458 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:10.736462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:10.736477 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:30:10.736528 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:10.736531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:10.736593 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:10.736686 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:10.737854 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:10.738613 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:10.739731 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:10.740119 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:10.740212 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:10.744654 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:10.744669 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:10.744685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:10.744717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:10.747646 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:10.748233 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:10.748241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:10.748243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:10.748356 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:10.748649 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:10.748652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:10.748654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:10.748963 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:10.749172 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:10.749361 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 21:30:10.749512 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":277,"threads_started":1}
W20260926 21:30:10.750641 2607 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53956
W20260926 21:30:10.751335 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:10.751545 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 2477
I20260926 21:30:10.776129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2095
I20260926 21:30:10.777431 2167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:10.895035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 2095
2026-09-26T21:30:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8532 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:10 dist-test-slave-87bt krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:10 dist-test-slave-87bt krb5kdc[2781](info): set up 2 sockets
Sep 26 21:30:10 dist-test-slave-87bt krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260926 21:30:11.990724 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:12 dist-test-slave-87bt krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:12.181764 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38627
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:12.510710 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:12.511000 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:12.511171 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:12.518306 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:12.518437 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:12.518563 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:12.518663 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:12.518747 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:12.529009 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:12.530296 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:12.531664 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:12.543197 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:12.543155 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:12.543120 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:12.544536 2794 server_base.cc:1034] running on GCE node
I20260926 21:30:12.545336 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:12.546756 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:12.547955 2794 hybrid_clock.cc:648] HybridClock initialized: now 1790458212547887 us; error 58 us; skew 500 ppm
I20260926 21:30:12.548298 2794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:12 dist-test-slave-87bt krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:12.551369 2794 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:30:12.552781 2794 webserver.cc:460] Webserver started at http://127.0.97.62:43909/ using document root <none> and password file <none>
I20260926 21:30:12.553396 2794 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:12.553491 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:12.553741 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:12.556546 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "7015c1db7c134f71be2f5a857bf93f13"
format_stamp: "Formatted at 2026-09-26 21:30:12 on dist-test-slave-87bt"
server_key: "0cbacabb3cd7ce0772153cd850f4af16"
server_key_iv: "c22f8aa4b171f99fc65fc8ca44c68acd"
server_key_version: "encryptionkey@0"
I20260926 21:30:12.557246 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "7015c1db7c134f71be2f5a857bf93f13"
format_stamp: "Formatted at 2026-09-26 21:30:12 on dist-test-slave-87bt"
server_key: "0cbacabb3cd7ce0772153cd850f4af16"
server_key_iv: "c22f8aa4b171f99fc65fc8ca44c68acd"
server_key_version: "encryptionkey@0"
I20260926 21:30:12.562098 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:30:12.565366 2809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:12.566713 2794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:30:12.566888 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "7015c1db7c134f71be2f5a857bf93f13"
format_stamp: "Formatted at 2026-09-26 21:30:12 on dist-test-slave-87bt"
server_key: "0cbacabb3cd7ce0772153cd850f4af16"
server_key_iv: "c22f8aa4b171f99fc65fc8ca44c68acd"
server_key_version: "encryptionkey@0"
I20260926 21:30:12.567090 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:12.588238 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:12.592299 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:12.592621 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:12.611209 2794 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38627
I20260926 21:30:12.611253 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38627 every 8 connection(s)
I20260926 21:30:12.612560 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:12.616142 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:12.619140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 2794
I20260926 21:30:12.619359 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:12.619951 388 external_mini_cluster.cc:1428] Setting key 2690e09116fde42d583f16f27ade853c
I20260926 21:30:12.628759 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13: Bootstrap starting.
I20260926 21:30:12.632467 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:12.633747 2861 log.cc:824] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:30:12 dist-test-slave-87bt krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:12.636601 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13: No bootstrap required, opened a new log
I20260926 21:30:12.643268 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:12.621927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.0.97.62:38627)
0926 21:30:12.622722 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:12.622737 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:12.622786 (+ 49us) server_negotiation.cc:407] Connection header received
0926 21:30:12.624627 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:12.624654 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:12.625167 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:12.625600 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:12.626895 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:12.628809 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:12.630666 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:12.631317 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:12.631474 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:12.636088 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:12.636095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:12.636109 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:12.636145 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:12.639255 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:12.640097 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:12.640102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:12.640108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:12.640271 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:12.640662 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:12.640665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:12.640667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:12.641147 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:12.641366 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:12.641798 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 21:30:12.642184 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":319,"threads_started":1}
I20260926 21:30:12.645531 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7015c1db7c134f71be2f5a857bf93f13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38627 } }
I20260926 21:30:12.645960 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:12.646085 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7015c1db7c134f71be2f5a857bf93f13, State: Initialized, Role: FOLLOWER
I20260926 21:30:12.646694 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7015c1db7c134f71be2f5a857bf93f13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38627 } }
I20260926 21:30:12.646895 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:12.646992 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:12.647104 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:12.649617 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7015c1db7c134f71be2f5a857bf93f13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38627 } }
I20260926 21:30:12.650262 2861 leader_election.cc:302] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7015c1db7c134f71be2f5a857bf93f13; no voters:
I20260926 21:30:12.650942 2861 leader_election.cc:288] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:12.651080 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:12.651852 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [term 1 LEADER]: Becoming Leader. State: Replica: 7015c1db7c134f71be2f5a857bf93f13, State: Running, Role: LEADER
I20260926 21:30:12.652447 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7015c1db7c134f71be2f5a857bf93f13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38627 } }
I20260926 21:30:12.652959 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:12.655414 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7015c1db7c134f71be2f5a857bf93f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7015c1db7c134f71be2f5a857bf93f13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38627 } } }
I20260926 21:30:12.655390 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7015c1db7c134f71be2f5a857bf93f13. Latest consensus state: current_term: 1 leader_uuid: "7015c1db7c134f71be2f5a857bf93f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7015c1db7c134f71be2f5a857bf93f13" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38627 } } }
I20260926 21:30:12.655826 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:12.655862 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:12.657680 2872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:12.662738 2872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:12.673631 2872 catalog_manager.cc:1347] Generated new cluster ID: 690ad1a047d54d1c92adb2fbec53f233
I20260926 21:30:12.673753 2872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:12.688544 2872 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:12.690330 2872 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:12.697952 2872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7015c1db7c134f71be2f5a857bf93f13: Generated new TSK 0
I20260926 21:30:12.698848 2872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:30:12.742051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38627
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:13.068056 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:13.068318 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:13.068437 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:13.075994 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:13.076102 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:13.076211 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:13.085883 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:13.087127 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:13.088328 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:13.101097 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:13.101265 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:13.101269 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:13.102463 2887 server_base.cc:1034] running on GCE node
I20260926 21:30:13.103087 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:13.103984 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:13.105216 2887 hybrid_clock.cc:648] HybridClock initialized: now 1790458213105171 us; error 31 us; skew 500 ppm
I20260926 21:30:13.105580 2887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:13 dist-test-slave-87bt krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:13.108826 2887 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:30:13.110322 2887 webserver.cc:460] Webserver started at http://127.0.97.1:33287/ using document root <none> and password file <none>
I20260926 21:30:13.111054 2887 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:13.111199 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:13.111519 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:13.114408 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "88b2459bdfe44943b344383c00394644"
format_stamp: "Formatted at 2026-09-26 21:30:13 on dist-test-slave-87bt"
server_key: "678ea4927aecd91988ae93247afd26bb"
server_key_iv: "16ddb54fe70b04c55e04e9b8753d247e"
server_key_version: "encryptionkey@0"
I20260926 21:30:13.115300 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "88b2459bdfe44943b344383c00394644"
format_stamp: "Formatted at 2026-09-26 21:30:13 on dist-test-slave-87bt"
server_key: "678ea4927aecd91988ae93247afd26bb"
server_key_iv: "16ddb54fe70b04c55e04e9b8753d247e"
server_key_version: "encryptionkey@0"
I20260926 21:30:13.120244 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:30:13.123757 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:13.125126 2887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:30:13.125294 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "88b2459bdfe44943b344383c00394644"
format_stamp: "Formatted at 2026-09-26 21:30:13 on dist-test-slave-87bt"
server_key: "678ea4927aecd91988ae93247afd26bb"
server_key_iv: "16ddb54fe70b04c55e04e9b8753d247e"
server_key_version: "encryptionkey@0"
I20260926 21:30:13.125500 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:13.147490 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:13.151747 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:13.152036 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:13.152988 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:13.154402 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:13.154567 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 21:30:13.154711 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:13.154781 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:13.187637 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44257
I20260926 21:30:13.187685 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44257 every 8 connection(s)
I20260926 21:30:13.188911 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:13.189438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 2887
I20260926 21:30:13.189697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:13.190517 388 external_mini_cluster.cc:1428] Setting key 4da48eb850c6f333a284b90e50d70c91
Sep 26 21:30:13 dist-test-slave-87bt krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:30:13.211750 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:13.191663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59915 (local address 127.0.97.62:38627)
0926 21:30:13.192068 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:13.192072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:13.193370 (+ 1298us) server_negotiation.cc:407] Connection header received
0926 21:30:13.195261 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:13.195269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:13.195357 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:13.195488 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:13.196913 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:13.197853 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:13.200527 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:13.200947 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:13.201053 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:13.205202 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:13.205211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:13.205219 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:13.205267 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:13.208808 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:13.209559 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:13.209565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:13.209568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:13.209709 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:13.210143 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:13.210149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:13.210153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:13.210541 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:13.210650 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:13.211338 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 21:30:13.211499 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":213,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:30:13.212736 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:13.192316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38627 (local address 127.0.97.1:59915)
0926 21:30:13.193206 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:13.193251 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:13.194996 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:13.195684 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:13.195692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:13.196193 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:13.196738 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:13.196753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:13.198017 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:13.198024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:13.200351 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:13.200372 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:13.201190 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:13.201199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:13.201536 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:13.202281 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:13.202313 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:13.204950 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:13.208957 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:13.208964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:13.208978 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:13.209396 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:13.209813 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:13.209817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:13.209819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:13.209991 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:13.210649 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:13.210655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:13.210981 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 21:30:13.211218 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 21:30:13.211612 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":392,"threads_started":1}
I20260926 21:30:13.214357 3015 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38627
I20260926 21:30:13.214692 3015 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:13.215387 3015 heartbeater.cc:511] Master 127.0.97.62:38627 requested a full tablet report, sending...
I20260926 21:30:13.217551 2826 ts_manager.cc:194] Registered new tserver with Master: 88b2459bdfe44943b344383c00394644 (127.0.97.1:44257)
I20260926 21:30:13.219128 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:30:13.235617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38627
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:13.550547 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:13.550822 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:13.550983 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:13.558259 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:13.558363 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:13.558548 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:13.568364 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:13.569623 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:13.571128 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:13.584302 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:13.584247 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:13.584267 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:13.585108 3021 server_base.cc:1034] running on GCE node
I20260926 21:30:13.585750 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:13.586838 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:13.588075 3021 hybrid_clock.cc:648] HybridClock initialized: now 1790458213587999 us; error 66 us; skew 500 ppm
I20260926 21:30:13.588419 3021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:13 dist-test-slave-87bt krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:13.591691 3021 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:30:13.593139 3021 webserver.cc:460] Webserver started at http://127.0.97.2:35903/ using document root <none> and password file <none>
I20260926 21:30:13.593825 3021 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:13.593974 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:13.594277 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:13.597172 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "680d4ca0380d4b91b54e6a666a2a5fca"
format_stamp: "Formatted at 2026-09-26 21:30:13 on dist-test-slave-87bt"
server_key: "1d3d6a869258c9c66978e62c0e7b82f2"
server_key_iv: "5de87c259d9e054835caf974aa886379"
server_key_version: "encryptionkey@0"
I20260926 21:30:13.597925 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "680d4ca0380d4b91b54e6a666a2a5fca"
format_stamp: "Formatted at 2026-09-26 21:30:13 on dist-test-slave-87bt"
server_key: "1d3d6a869258c9c66978e62c0e7b82f2"
server_key_iv: "5de87c259d9e054835caf974aa886379"
server_key_version: "encryptionkey@0"
I20260926 21:30:13.602900 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:30:13.606140 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:13.607511 3021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:13.607704 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "680d4ca0380d4b91b54e6a666a2a5fca"
format_stamp: "Formatted at 2026-09-26 21:30:13 on dist-test-slave-87bt"
server_key: "1d3d6a869258c9c66978e62c0e7b82f2"
server_key_iv: "5de87c259d9e054835caf974aa886379"
server_key_version: "encryptionkey@0"
I20260926 21:30:13.607931 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:13.631359 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:13.635898 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:13.636134 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:13.637001 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:13.638309 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:13.638384 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:13.638535 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:13.638585 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:13.671819 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42931
I20260926 21:30:13.671854 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42931 every 8 connection(s)
I20260926 21:30:13.673367 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:13.683514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 3021
I20260926 21:30:13.683725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:13.684253 388 external_mini_cluster.cc:1428] Setting key 371740acb872e3ec4352cc062451a8d8
Sep 26 21:30:13 dist-test-slave-87bt krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:13.696107 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:13.676086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54081 (local address 127.0.97.62:38627)
0926 21:30:13.676300 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:13.676307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:13.677533 (+ 1226us) server_negotiation.cc:407] Connection header received
0926 21:30:13.679517 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:13.679522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:13.679599 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:13.679700 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:13.681614 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:13.682403 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:13.685200 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:13.685759 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:13.685900 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:13.690200 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:13.690206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:13.690209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:13.690243 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:13.692943 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:13.693740 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:13.693749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:13.693752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:13.693886 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:13.694319 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:13.694326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:13.694330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:13.694658 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:13.694824 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:13.695528 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 21:30:13.695757 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:30:13.697238 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:13.676591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38627 (local address 127.0.97.2:54081)
0926 21:30:13.677331 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:13.677394 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:13.679197 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:13.680004 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:13.680014 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:13.680703 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:13.681357 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:13.681387 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:13.682661 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:13.682665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:13.684863 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:13.684885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:13.686003 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:13.686008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:13.686309 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:13.687111 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:13.687150 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:13.689928 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:13.693117 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:13.693127 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:13.693148 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:13.693605 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:13.694006 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:13.694011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:13.694016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:13.694209 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:13.694788 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:13.694796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:13.695164 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 21:30:13.695447 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 21:30:13.696021 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":293,"threads_started":1}
I20260926 21:30:13.698951 3149 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38627
I20260926 21:30:13.699273 3149 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:13.700008 3149 heartbeater.cc:511] Master 127.0.97.62:38627 requested a full tablet report, sending...
I20260926 21:30:13.701588 2826 ts_manager.cc:194] Registered new tserver with Master: 680d4ca0380d4b91b54e6a666a2a5fca (127.0.97.2:42931)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:30:13.702708 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:30:13.729579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38627
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:14.057545 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:14.057852 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:14.057945 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:14.065887 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:14.066025 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:14.066139 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:14.076169 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:14.077407 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:14.078662 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:14.092090 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:14.092116 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:14.092137 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:14.092758 3154 server_base.cc:1034] running on GCE node
I20260926 21:30:14.093482 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:14.094476 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:14.095690 3154 hybrid_clock.cc:648] HybridClock initialized: now 1790458214095650 us; error 38 us; skew 500 ppm
I20260926 21:30:14.095965 3154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:14 dist-test-slave-87bt krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:14.099272 3154 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:30:14.100646 3154 webserver.cc:460] Webserver started at http://127.0.97.3:38691/ using document root <none> and password file <none>
I20260926 21:30:14.101336 3154 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:14.101480 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:14.101822 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:14.104764 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "1f9a8dec97ce4419945ec9fcf699f53e"
format_stamp: "Formatted at 2026-09-26 21:30:14 on dist-test-slave-87bt"
server_key: "b6047bf305c44e112fed23a16ba29573"
server_key_iv: "25c0f6d2716f639515cd655a181db4e5"
server_key_version: "encryptionkey@0"
I20260926 21:30:14.105547 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1f9a8dec97ce4419945ec9fcf699f53e"
format_stamp: "Formatted at 2026-09-26 21:30:14 on dist-test-slave-87bt"
server_key: "b6047bf305c44e112fed23a16ba29573"
server_key_iv: "25c0f6d2716f639515cd655a181db4e5"
server_key_version: "encryptionkey@0"
I20260926 21:30:14.110698 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 21:30:14.114136 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:14.115520 3154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:14.115697 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "1f9a8dec97ce4419945ec9fcf699f53e"
format_stamp: "Formatted at 2026-09-26 21:30:14 on dist-test-slave-87bt"
server_key: "b6047bf305c44e112fed23a16ba29573"
server_key_iv: "25c0f6d2716f639515cd655a181db4e5"
server_key_version: "encryptionkey@0"
I20260926 21:30:14.115919 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:14.132230 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:14.136652 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:14.136955 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:14.137935 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:14.139324 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:14.139428 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:14.139580 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:14.139652 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:14.172264 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36627
I20260926 21:30:14.172302 3281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36627 every 8 connection(s)
I20260926 21:30:14.173586 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:14.178285 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 3154
I20260926 21:30:14.178539 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:14.179014 388 external_mini_cluster.cc:1428] Setting key 9c2e51d92fee643b05c7098b4188bf59
Sep 26 21:30:14 dist-test-slave-87bt krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:14.195745 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:14.176578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56617 (local address 127.0.97.62:38627)
0926 21:30:14.176808 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:14.176815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:14.178089 (+ 1274us) server_negotiation.cc:407] Connection header received
0926 21:30:14.180328 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:14.180334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:14.180417 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:14.180537 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:14.182351 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:14.183166 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:14.184727 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:14.185130 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:14.185414 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:14.189353 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:14.189359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:14.189362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:14.189400 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:14.192374 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:14.193303 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:14.193316 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:14.193320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:14.193481 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:14.194059 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:14.194066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:14.194070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:14.194362 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:14.194541 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:14.195332 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 21:30:14.195508 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 21:30:14.196704 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:14.177188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38627 (local address 127.0.97.3:56617)
0926 21:30:14.177922 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:14.177960 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:14.180015 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:14.180786 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:14.180794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:14.181504 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:14.182189 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:14.182206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:14.183333 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:14.183337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:14.184573 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:14.184585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:14.185510 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:14.185519 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:14.185771 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:14.186601 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:14.186626 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:14.189082 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:14.192551 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:14.192560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:14.192578 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:14.193166 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:14.193623 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:14.193628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:14.193631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:14.193929 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:14.194533 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:14.194541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:14.194970 (+ 429us) client_negotiation.cc:769] Sending connection context
0926 21:30:14.195221 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 21:30:14.195615 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":302,"threads_started":1}
I20260926 21:30:14.198057 3282 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38627
I20260926 21:30:14.198361 3282 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:14.199136 3282 heartbeater.cc:511] Master 127.0.97.62:38627 requested a full tablet report, sending...
I20260926 21:30:14.200656 2826 ts_manager.cc:194] Registered new tserver with Master: 1f9a8dec97ce4419945ec9fcf699f53e (127.0.97.3:36627)
I20260926 21:30:14.201414 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56617
I20260926 21:30:14.207172 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:30:14.222020 3015 heartbeater.cc:503] Master 127.0.97.62:38627 was elected leader, sending a full tablet report...
W20260926 21:30:14.238250 3017 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:14.235881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.0.97.62:38627)
0926 21:30:14.236126 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:14.236136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:14.236452 (+ 316us) server_negotiation.cc:407] Connection header received
0926 21:30:14.236592 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:14.236596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:14.236657 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:14.236746 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:14.237975 (+ 1229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35242: BlockingRecv error: recv got EOF from 127.0.0.1:35242 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:30:14.241575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2887
W20260926 21:30:14.244350 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2887: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:44257: server requires authentication, but client does not have Kerberos credentials available
I20260926 21:30:14.244536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 2887
I20260926 21:30:14.260352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3021
W20260926 21:30:14.263037 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3021: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:42931: server requires authentication, but client does not have Kerberos credentials available
I20260926 21:30:14.263207 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 3021
I20260926 21:30:14.278303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3154
W20260926 21:30:14.281009 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3154: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:36627: server requires authentication, but client does not have Kerberos credentials available
I20260926 21:30:14.281150 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 3154
I20260926 21:30:14.296172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 2794
I20260926 21:30:14.297375 2856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:14.397584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 2794
2026-09-26T21:30:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3499 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:14 dist-test-slave-87bt krb5kdc[3299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:14 dist-test-slave-87bt krb5kdc[3299](info): set up 2 sockets
Sep 26 21:30:14 dist-test-slave-87bt krb5kdc[3299](info): commencing operation
krb5kdc: starting...
W20260926 21:30:15.474792 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:15 dist-test-slave-87bt krb5kdc[3299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:15.665900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33909
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:15.981146 3312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:15.981509 3312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:15.981679 3312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:15.988796 3312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:15.988973 3312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:15.989087 3312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:15.989192 3312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:15.989289 3312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:15.999316 3312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:16.000653 3312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:16.001964 3312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:16.013753 3317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:16.013823 3320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:16.013845 3318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:16.014709 3312 server_base.cc:1034] running on GCE node
I20260926 21:30:16.015486 3312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:16.016901 3312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:16.018131 3312 hybrid_clock.cc:648] HybridClock initialized: now 1790458216018086 us; error 44 us; skew 500 ppm
I20260926 21:30:16.018538 3312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:16 dist-test-slave-87bt krb5kdc[3299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:16.025543 3312 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:30:16.027269 3312 webserver.cc:460] Webserver started at http://127.0.97.62:38233/ using document root <none> and password file <none>
I20260926 21:30:16.027983 3312 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:16.028133 3312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:16.028453 3312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:16.031596 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "506bd7dc18574c809c46b6e719612882"
format_stamp: "Formatted at 2026-09-26 21:30:16 on dist-test-slave-87bt"
server_key: "1dab085820cbb547d2ce4d8529ba4ae0"
server_key_iv: "3e600c29f441a668e7b0163dbfc94e1c"
server_key_version: "encryptionkey@0"
I20260926 21:30:16.032457 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "506bd7dc18574c809c46b6e719612882"
format_stamp: "Formatted at 2026-09-26 21:30:16 on dist-test-slave-87bt"
server_key: "1dab085820cbb547d2ce4d8529ba4ae0"
server_key_iv: "3e600c29f441a668e7b0163dbfc94e1c"
server_key_version: "encryptionkey@0"
I20260926 21:30:16.038053 3312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:30:16.041887 3327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:16.043488 3312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:16.043694 3312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "506bd7dc18574c809c46b6e719612882"
format_stamp: "Formatted at 2026-09-26 21:30:16 on dist-test-slave-87bt"
server_key: "1dab085820cbb547d2ce4d8529ba4ae0"
server_key_iv: "3e600c29f441a668e7b0163dbfc94e1c"
server_key_version: "encryptionkey@0"
I20260926 21:30:16.043922 3312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:16.068361 3312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:16.072737 3312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:16.073037 3312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:16.092296 3312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33909
I20260926 21:30:16.092324 3378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33909 every 8 connection(s)
I20260926 21:30:16.093721 3312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:16.097615 3379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:16.103729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 3312
I20260926 21:30:16.103955 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:16.104580 388 external_mini_cluster.cc:1428] Setting key 378122720ae19f6df8e467af039060ca
I20260926 21:30:16.109848 3379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882: Bootstrap starting.
I20260926 21:30:16.113902 3379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:16.115301 3379 log.cc:824] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:16.118705 3379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882: No bootstrap required, opened a new log
Sep 26 21:30:16 dist-test-slave-87bt krb5kdc[3299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:16.132400 3379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506bd7dc18574c809c46b6e719612882" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33909 } }
I20260926 21:30:16.132936 3379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:16.133057 3379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506bd7dc18574c809c46b6e719612882, State: Initialized, Role: FOLLOWER
I20260926 21:30:16.133862 3379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506bd7dc18574c809c46b6e719612882" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33909 } }
I20260926 21:30:16.134155 3379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:16.134323 3379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:16.134606 3379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:16.137167 3379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506bd7dc18574c809c46b6e719612882" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33909 } }
I20260926 21:30:16.137782 3379 leader_election.cc:302] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 506bd7dc18574c809c46b6e719612882; no voters:
I20260926 21:30:16.138545 3379 leader_election.cc:288] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:16.138749 3384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:16.139094 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:16.106644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.97.62:33909)
0926 21:30:16.107532 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:16.107550 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:16.107607 (+ 57us) server_negotiation.cc:407] Connection header received
0926 21:30:16.110083 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:16.110115 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:16.110754 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:16.111235 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:16.112733 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:16.114811 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:16.116481 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:16.117586 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:16.117777 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:16.121767 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:16.121780 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:16.121806 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:16.121871 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:16.131573 (+ 9702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:16.132580 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:16.132593 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:16.132606 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:16.132848 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:16.133415 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:16.133423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:16.133428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:16.134510 (+ 1082us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:16.134888 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:16.136629 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0926 21:30:16.137312 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":361,"threads_started":1}
I20260926 21:30:16.139969 3384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [term 1 LEADER]: Becoming Leader. State: Replica: 506bd7dc18574c809c46b6e719612882, State: Running, Role: LEADER
I20260926 21:30:16.140853 3384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506bd7dc18574c809c46b6e719612882" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33909 } }
I20260926 21:30:16.144428 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:30:16.147814 3379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:16.155954 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "506bd7dc18574c809c46b6e719612882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506bd7dc18574c809c46b6e719612882" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33909 } } }
I20260926 21:30:16.156023 3385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 506bd7dc18574c809c46b6e719612882. Latest consensus state: current_term: 1 leader_uuid: "506bd7dc18574c809c46b6e719612882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506bd7dc18574c809c46b6e719612882" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33909 } } }
I20260926 21:30:16.156363 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:16.156416 3385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:16.157269 3394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:16.170400 3394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:16.185232 3394 catalog_manager.cc:1347] Generated new cluster ID: 5cbbb7ee375c4ca28bb2d04c5d062162
I20260926 21:30:16.185420 3394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:16.218573 3394 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:16.221185 3394 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:16.232235 3394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 506bd7dc18574c809c46b6e719612882: Generated new TSK 0
I20260926 21:30:16.233633 3394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:30:16.247601 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:16.238508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.0.97.62:33909)
0926 21:30:16.238693 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:16.238700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:16.239372 (+ 672us) server_negotiation.cc:407] Connection header received
0926 21:30:16.239704 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:16.239712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:16.239804 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:16.239941 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:16.241096 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:16.242011 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:16.243874 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:16.244597 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:16.244757 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:16.245968 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:16.245973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:16.245976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:16.246708 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:16.246785 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:16.246809 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:16.246992 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:16.247079 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 21:30:16.247183 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 21:30:16.254454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3312
I20260926 21:30:16.255716 3374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:16.342584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 3312
2026-09-26T21:30:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1939 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:16 dist-test-slave-87bt krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:16 dist-test-slave-87bt krb5kdc[3414](info): set up 2 sockets
Sep 26 21:30:16 dist-test-slave-87bt krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260926 21:30:17.426741 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:17 dist-test-slave-87bt krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:17.618076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44545
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:17.938568 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:17.938915 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:17.939074 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:17.946272 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:17.946457 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:17.946560 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:17.946660 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:17.946735 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:17.956571 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:17.957902 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:17.959270 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:17.970767 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:17.970880 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:17.970894 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:17.971710 3427 server_base.cc:1034] running on GCE node
I20260926 21:30:17.972441 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:17.973732 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:17.974979 3427 hybrid_clock.cc:648] HybridClock initialized: now 1790458217974940 us; error 44 us; skew 500 ppm
I20260926 21:30:17.975349 3427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:17 dist-test-slave-87bt krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:17.978529 3427 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:30:17.980118 3427 webserver.cc:460] Webserver started at http://127.0.97.62:35531/ using document root <none> and password file <none>
I20260926 21:30:17.980834 3427 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:17.980974 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:17.981288 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:17.984318 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "c2e45aaf2f39481381c5fc645519a578"
format_stamp: "Formatted at 2026-09-26 21:30:17 on dist-test-slave-87bt"
server_key: "18884eb2fc6cdf44331a0e9320884b9e"
server_key_iv: "f56c67ca111ad306f4051e878f4c23bc"
server_key_version: "encryptionkey@0"
I20260926 21:30:17.985083 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "c2e45aaf2f39481381c5fc645519a578"
format_stamp: "Formatted at 2026-09-26 21:30:17 on dist-test-slave-87bt"
server_key: "18884eb2fc6cdf44331a0e9320884b9e"
server_key_iv: "f56c67ca111ad306f4051e878f4c23bc"
server_key_version: "encryptionkey@0"
I20260926 21:30:17.990172 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 21:30:17.993520 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:17.995054 3427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:17.995218 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "c2e45aaf2f39481381c5fc645519a578"
format_stamp: "Formatted at 2026-09-26 21:30:17 on dist-test-slave-87bt"
server_key: "18884eb2fc6cdf44331a0e9320884b9e"
server_key_iv: "f56c67ca111ad306f4051e878f4c23bc"
server_key_version: "encryptionkey@0"
I20260926 21:30:17.995467 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:18.007887 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:18.011755 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:18.012077 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:18.030470 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44545
I20260926 21:30:18.030511 3493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44545 every 8 connection(s)
I20260926 21:30:18.031842 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:18.035427 3494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:18.036077 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 3427
I20260926 21:30:18.036283 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:18.036906 388 external_mini_cluster.cc:1428] Setting key 32a26498d646f56e193024b90aa261b4
I20260926 21:30:18.049754 3494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578: Bootstrap starting.
Sep 26 21:30:18 dist-test-slave-87bt krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:18.052954 3494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:18.054376 3494 log.cc:824] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:18.057760 3494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578: No bootstrap required, opened a new log
I20260926 21:30:18.061199 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:18.038939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.0.97.62:44545)
0926 21:30:18.039794 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:18.039810 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:18.039863 (+ 53us) server_negotiation.cc:407] Connection header received
0926 21:30:18.041999 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:18.042031 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:18.042642 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:18.043140 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:18.044396 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:18.046407 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:18.047652 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:18.048718 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:18.048890 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:18.052798 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:18.052811 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:18.052837 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:18.052886 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:18.056641 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:18.057314 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:18.057322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:18.057331 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:18.057525 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:18.057871 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:18.057876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:18.057879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:18.058629 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:18.058920 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:18.059311 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 21:30:18.059748 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":378,"threads_started":1}
I20260926 21:30:18.066797 3494 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e45aaf2f39481381c5fc645519a578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44545 } }
I20260926 21:30:18.067106 3494 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:18.067179 3494 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e45aaf2f39481381c5fc645519a578, State: Initialized, Role: FOLLOWER
I20260926 21:30:18.067641 3494 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e45aaf2f39481381c5fc645519a578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44545 } }
I20260926 21:30:18.067816 3494 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:18.067904 3494 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:18.068011 3494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:18.070094 3494 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e45aaf2f39481381c5fc645519a578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44545 } }
I20260926 21:30:18.070729 3494 leader_election.cc:302] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2e45aaf2f39481381c5fc645519a578; no voters:
I20260926 21:30:18.071266 3494 leader_election.cc:288] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:18.071394 3499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:18.072059 3499 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [term 1 LEADER]: Becoming Leader. State: Replica: c2e45aaf2f39481381c5fc645519a578, State: Running, Role: LEADER
I20260926 21:30:18.072585 3499 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e45aaf2f39481381c5fc645519a578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44545 } }
I20260926 21:30:18.073201 3494 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:18.075572 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2e45aaf2f39481381c5fc645519a578. Latest consensus state: current_term: 1 leader_uuid: "c2e45aaf2f39481381c5fc645519a578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e45aaf2f39481381c5fc645519a578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44545 } } }
I20260926 21:30:18.075970 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:18.077292 3506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:18.077169 3500 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2e45aaf2f39481381c5fc645519a578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e45aaf2f39481381c5fc645519a578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44545 } } }
I20260926 21:30:18.077669 3500 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:18.082809 3506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:18.094244 3506 catalog_manager.cc:1347] Generated new cluster ID: 2fcbd09ff16f4aa09b2ca48b7b6fcf49
I20260926 21:30:18.094496 3506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:18.118826 3506 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:18.120201 3506 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:18.132069 3506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2e45aaf2f39481381c5fc645519a578: Generated new TSK 0
I20260926 21:30:18.132921 3506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:30:18.191226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44545
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:18.528174 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:18.528475 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:18.528573 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:18.535792 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:18.535900 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:18.536012 3520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:18.545974 3520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:18.547250 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:18.548476 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:18.561478 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:18.561364 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:18.561395 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:18.563814 3520 server_base.cc:1034] running on GCE node
I20260926 21:30:18.564376 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:18.565292 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:18.566552 3520 hybrid_clock.cc:648] HybridClock initialized: now 1790458218566504 us; error 45 us; skew 500 ppm
I20260926 21:30:18.566892 3520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:18 dist-test-slave-87bt krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:18.570134 3520 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:30:18.571759 3520 webserver.cc:460] Webserver started at http://127.0.97.1:33589/ using document root <none> and password file <none>
I20260926 21:30:18.572494 3520 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:18.572638 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:18.572927 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:18.575907 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "93ebe791f364417488d21580ff5d8b86"
format_stamp: "Formatted at 2026-09-26 21:30:18 on dist-test-slave-87bt"
server_key: "499200cb002da396c8fbb9788b86fc70"
server_key_iv: "5ff59a0bb53691b877da97cf4b420f61"
server_key_version: "encryptionkey@0"
I20260926 21:30:18.576668 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "93ebe791f364417488d21580ff5d8b86"
format_stamp: "Formatted at 2026-09-26 21:30:18 on dist-test-slave-87bt"
server_key: "499200cb002da396c8fbb9788b86fc70"
server_key_iv: "5ff59a0bb53691b877da97cf4b420f61"
server_key_version: "encryptionkey@0"
I20260926 21:30:18.581995 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:30:18.585325 3535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:18.586678 3520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:30:18.586843 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "93ebe791f364417488d21580ff5d8b86"
format_stamp: "Formatted at 2026-09-26 21:30:18 on dist-test-slave-87bt"
server_key: "499200cb002da396c8fbb9788b86fc70"
server_key_iv: "5ff59a0bb53691b877da97cf4b420f61"
server_key_version: "encryptionkey@0"
I20260926 21:30:18.587033 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:18.603796 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:18.607728 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:18.608023 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:18.608961 3520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:18.610314 3520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:18.610419 3520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:18.610612 3520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:18.610685 3520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:18.643405 3520 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46835
I20260926 21:30:18.643448 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46835 every 8 connection(s)
I20260926 21:30:18.644690 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:18.652482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 3520
I20260926 21:30:18.652683 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:18.653411 388 external_mini_cluster.cc:1428] Setting key 63b82ae12a0789bce2d19352a1acd65a
Sep 26 21:30:18 dist-test-slave-87bt krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:18.665055 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:18.647267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55937 (local address 127.0.97.62:44545)
0926 21:30:18.647712 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:18.647717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:18.648558 (+ 841us) server_negotiation.cc:407] Connection header received
0926 21:30:18.650304 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:18.650309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:18.650398 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:18.650561 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:18.652016 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:18.653148 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:18.655016 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:18.655474 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:18.655577 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:18.659662 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:18.659667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:18.659670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:18.659700 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:18.662101 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:18.662866 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:18.662875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:18.662879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:18.663001 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:18.663419 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:18.663424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:18.663427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:18.663673 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:18.663822 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:18.664491 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 21:30:18.664700 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":256,"threads_started":1}
I20260926 21:30:18.665998 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:18.647722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44545 (local address 127.0.97.1:55937)
0926 21:30:18.648392 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:18.648438 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:18.650047 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:18.650795 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:18.650803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:18.651295 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:18.651843 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:18.651859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:18.653309 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:18.653314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:18.654822 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:18.654834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:18.655704 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:18.655711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:18.656034 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:18.656743 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:18.656778 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:18.659431 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:18.662242 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:18.662251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:18.662267 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:18.662749 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:18.663117 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:18.663122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:18.663126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:18.663302 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:18.663796 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:18.663804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:18.664133 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 21:30:18.664386 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 21:30:18.664893 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":261,"threads_started":1}
I20260926 21:30:18.667654 3648 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44545
I20260926 21:30:18.667937 3648 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:18.668648 3648 heartbeater.cc:511] Master 127.0.97.62:44545 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:30:18.670728 3459 ts_manager.cc:194] Registered new tserver with Master: 93ebe791f364417488d21580ff5d8b86 (127.0.97.1:46835)
I20260926 21:30:18.672247 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:30:18.697549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44545
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:19.020872 3654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:19.021144 3654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:19.021236 3654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:19.028383 3654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:19.028501 3654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:19.028610 3654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:19.038385 3654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:19.039786 3654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:19.041126 3654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:19.054416 3659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:19.054513 3660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:19.054708 3662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:19.055549 3654 server_base.cc:1034] running on GCE node
I20260926 21:30:19.056188 3654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:19.057130 3654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:19.058350 3654 hybrid_clock.cc:648] HybridClock initialized: now 1790458219058285 us; error 64 us; skew 500 ppm
I20260926 21:30:19.058684 3654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:19.061843 3654 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:30:19.063298 3654 webserver.cc:460] Webserver started at http://127.0.97.2:37621/ using document root <none> and password file <none>
I20260926 21:30:19.064003 3654 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:19.064144 3654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:19.064441 3654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:19.067432 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "d8662cd17a374004aad909c0c13f0798"
format_stamp: "Formatted at 2026-09-26 21:30:19 on dist-test-slave-87bt"
server_key: "a1cd57b56ed1677898150c7cf898d207"
server_key_iv: "3d7d6226a6f80cbcee192d777e649388"
server_key_version: "encryptionkey@0"
I20260926 21:30:19.068233 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d8662cd17a374004aad909c0c13f0798"
format_stamp: "Formatted at 2026-09-26 21:30:19 on dist-test-slave-87bt"
server_key: "a1cd57b56ed1677898150c7cf898d207"
server_key_iv: "3d7d6226a6f80cbcee192d777e649388"
server_key_version: "encryptionkey@0"
I20260926 21:30:19.073414 3654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:30:19.076977 3669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:19.078553 3654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:19.078720 3654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "d8662cd17a374004aad909c0c13f0798"
format_stamp: "Formatted at 2026-09-26 21:30:19 on dist-test-slave-87bt"
server_key: "a1cd57b56ed1677898150c7cf898d207"
server_key_iv: "3d7d6226a6f80cbcee192d777e649388"
server_key_version: "encryptionkey@0"
I20260926 21:30:19.078953 3654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:19.103085 3654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:19.107751 3654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:19.108073 3654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:19.109066 3654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:19.110476 3654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:19.110584 3654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:19.110733 3654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:19.110807 3654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:19.145295 3654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46109
I20260926 21:30:19.145445 3781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46109 every 8 connection(s)
I20260926 21:30:19.146687 3654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:19.155942 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 3654
I20260926 21:30:19.156127 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:19.156555 388 external_mini_cluster.cc:1428] Setting key 8be77d9f44fb4d52b23f2656d2b2f82d
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:19.168483 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:19.149325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53049 (local address 127.0.97.62:44545)
0926 21:30:19.149509 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:19.149514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:19.150820 (+ 1306us) server_negotiation.cc:407] Connection header received
0926 21:30:19.152652 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:19.152657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:19.152730 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:19.152828 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:19.154396 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.155180 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.156936 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.157398 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.157549 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:19.161420 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:19.161426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:19.161429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:19.161465 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:19.164850 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.165782 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.165791 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.165797 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.165976 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.166599 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.166608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.166613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.166945 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:19.167113 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:19.167888 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 21:30:19.168127 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:30:19.169560 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:19.149792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44545 (local address 127.0.97.2:53049)
0926 21:30:19.150639 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:19.150686 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:19.152332 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:19.153066 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:19.153075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:19.153620 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:19.154216 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:19.154233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.155309 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.155312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:19.156779 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:19.156792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.157645 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.157652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:19.157898 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:19.158671 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:19.158694 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:19.161134 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:19.165094 (+ 3960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:19.165102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:19.165118 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:19.165634 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:19.166127 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:19.166131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:19.166134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:19.166365 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:19.167111 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:19.167121 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:19.167485 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 21:30:19.167779 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 21:30:19.168374 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":313,"threads_started":1}
I20260926 21:30:19.171417 3782 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44545
I20260926 21:30:19.171753 3782 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:19.172557 3782 heartbeater.cc:511] Master 127.0.97.62:44545 requested a full tablet report, sending...
I20260926 21:30:19.174499 3459 ts_manager.cc:194] Registered new tserver with Master: d8662cd17a374004aad909c0c13f0798 (127.0.97.2:46109)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:30:19.175639 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:30:19.202888 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44545
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:19.533428 3787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:19.533761 3787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:19.533928 3787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:19.541173 3787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:19.541318 3787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:19.541487 3787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:19.551106 3787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:19.552455 3787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:19.553774 3787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:19.566828 3795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:19.566920 3793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:19.566901 3792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:19.567673 3787 server_base.cc:1034] running on GCE node
I20260926 21:30:19.568322 3787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:19.569229 3787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:19.570466 3787 hybrid_clock.cc:648] HybridClock initialized: now 1790458219570385 us; error 29 us; skew 500 ppm
I20260926 21:30:19.570811 3787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:19.573809 3787 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:30:19.575309 3787 webserver.cc:460] Webserver started at http://127.0.97.3:35731/ using document root <none> and password file <none>
I20260926 21:30:19.575996 3787 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:19.576138 3787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:19.576435 3787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:19.579562 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "82be803b0bba4c7e8545f13daa06d71e"
format_stamp: "Formatted at 2026-09-26 21:30:19 on dist-test-slave-87bt"
server_key: "c6680e6ce23eaa846a5197a69fc3f8a0"
server_key_iv: "72020247ad682258381836caed57f18d"
server_key_version: "encryptionkey@0"
I20260926 21:30:19.580333 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "82be803b0bba4c7e8545f13daa06d71e"
format_stamp: "Formatted at 2026-09-26 21:30:19 on dist-test-slave-87bt"
server_key: "c6680e6ce23eaa846a5197a69fc3f8a0"
server_key_iv: "72020247ad682258381836caed57f18d"
server_key_version: "encryptionkey@0"
I20260926 21:30:19.585606 3787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 21:30:19.589012 3802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:19.590291 3787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:30:19.590524 3787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "82be803b0bba4c7e8545f13daa06d71e"
format_stamp: "Formatted at 2026-09-26 21:30:19 on dist-test-slave-87bt"
server_key: "c6680e6ce23eaa846a5197a69fc3f8a0"
server_key_iv: "72020247ad682258381836caed57f18d"
server_key_version: "encryptionkey@0"
I20260926 21:30:19.590729 3787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:19.607671 3787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:19.611881 3787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:19.612135 3787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:19.613059 3787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:19.614337 3787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:19.614502 3787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:19.614611 3787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:19.614648 3787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:19.647843 3787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38225
I20260926 21:30:19.647881 3914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38225 every 8 connection(s)
I20260926 21:30:19.649120 3787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:19.652274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 3787
I20260926 21:30:19.652410 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:19.652871 388 external_mini_cluster.cc:1428] Setting key ec422446c81480ae407bbd8cb5e9d28a
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:19.668324 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:19.651734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44549 (local address 127.0.97.62:44545)
0926 21:30:19.651957 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:19.651963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:19.653363 (+ 1400us) server_negotiation.cc:407] Connection header received
0926 21:30:19.655164 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:19.655169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:19.655243 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:19.655347 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:19.656777 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.657507 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.659030 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.659398 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.659485 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:19.662762 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:19.662767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:19.662770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:19.662799 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:19.665048 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.665732 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.665735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.665738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.665868 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.666541 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.666545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.666549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.666837 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:19.667007 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:19.667781 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 21:30:19.667999 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:30:19.669165 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:19.652294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44545 (local address 127.0.97.3:44549)
0926 21:30:19.653187 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:19.653233 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:19.654923 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:19.655613 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:19.655620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:19.656104 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:19.656639 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:19.656654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.657643 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.657647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:19.658916 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:19.658928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.659572 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.659577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:19.659755 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:19.660421 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:19.660446 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:19.662583 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:19.665182 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:19.665189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:19.665201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:19.665625 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:19.665985 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:19.665990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:19.665995 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:19.666174 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:19.666983 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:19.666991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:19.667308 (+ 317us) client_negotiation.cc:769] Sending connection context
0926 21:30:19.667573 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 21:30:19.668073 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":370,"threads_started":1}
I20260926 21:30:19.670517 3915 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44545
I20260926 21:30:19.670856 3915 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:19.671525 3915 heartbeater.cc:511] Master 127.0.97.62:44545 requested a full tablet report, sending...
I20260926 21:30:19.672791 3459 ts_manager.cc:194] Registered new tserver with Master: 82be803b0bba4c7e8545f13daa06d71e (127.0.97.3:38225)
I20260926 21:30:19.673593 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44549
I20260926 21:30:19.675331 3648 heartbeater.cc:503] Master 127.0.97.62:44545 was elected leader, sending a full tablet report...
I20260926 21:30:19.679939 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:19.728514 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:19.714348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.0.97.62:44545)
0926 21:30:19.714595 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:19.714605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:19.715068 (+ 463us) server_negotiation.cc:407] Connection header received
0926 21:30:19.715211 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:19.715216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:19.715278 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:19.715389 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:19.716762 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.717477 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.718775 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.719169 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.719267 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:19.723300 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:19.723305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:19.723309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:19.723347 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:19.726021 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.726851 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.726857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.726859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.726998 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.727336 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.727339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.727344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.727669 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:19.727849 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:19.728107 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 21:30:19.728269 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260926 21:30:19.729032 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34050
I20260926 21:30:19.730659 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3520
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:19.746294 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:19.731844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.0.97.1:46835)
0926 21:30:19.732489 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:19.732496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:19.732522 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:30:19.732576 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:19.732583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:19.732805 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:19.732933 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:19.734073 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.735179 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.736308 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.736863 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.736973 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:19.740882 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:19.740895 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:19.740918 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:19.740952 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:19.743688 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.744349 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.744356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.744358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.744472 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.744771 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.744774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.744776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.745102 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:19.745315 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:19.745610 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 21:30:19.745894 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":325,"threads_started":1}
W20260926 21:30:19.747623 3643 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54564
W20260926 21:30:19.748804 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:19.749029 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 3520
I20260926 21:30:19.765687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3654
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:30:19.781282 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:19.766646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.0.97.2:46109)
0926 21:30:19.767207 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:19.767214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:19.767239 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:30:19.767296 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:19.767303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:19.767513 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:19.767642 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:19.768772 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.769752 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.771002 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.771535 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.771632 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:19.775691 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:19.775704 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:19.775719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:19.775761 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:19.778656 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.779316 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.779324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.779326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.779449 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.779766 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.779769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.779771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.780101 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:19.780324 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:19.780687 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 21:30:19.780972 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":307,"threads_started":1}
W20260926 21:30:19.782783 3777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57944
W20260926 21:30:19.783924 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3654: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:19.784091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 3654
I20260926 21:30:19.799835 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3787
Sep 26 21:30:19 dist-test-slave-87bt krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458219, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:30:19.816095 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:19.800828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.97.3:38225)
0926 21:30:19.801529 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:19.801536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:19.801565 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:30:19.801628 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:19.801634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:19.801831 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:19.801979 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:19.803332 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.804383 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.805664 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:19.806275 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:19.806381 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:19.810907 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:19.810921 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:19.810938 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:19.810975 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:19.813531 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.814154 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.814161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.814163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.814277 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:19.814719 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:19.814722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:19.814724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:19.815046 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:19.815259 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:19.815531 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 21:30:19.815716 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":390,"threads_started":1}
W20260926 21:30:19.817330 3910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33424
W20260926 21:30:19.818470 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3787: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:30:19.818670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 3787
I20260926 21:30:19.833887 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3427
I20260926 21:30:19.835083 3489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:19.934945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 3427
2026-09-26T21:30:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3598 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:20 dist-test-slave-87bt krb5kdc[3932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:20 dist-test-slave-87bt krb5kdc[3932](info): set up 2 sockets
Sep 26 21:30:20 dist-test-slave-87bt krb5kdc[3932](info): commencing operation
krb5kdc: starting...
W20260926 21:30:21.098855 388 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:21 dist-test-slave-87bt krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:21.294361 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45469
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:21.614197 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:21.614575 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:21.614715 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:21.622222 3945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:21.622331 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:21.622411 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:21.622527 3945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:21.622610 3945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:21.632634 3945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:21.633985 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:21.635416 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:21.647037 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:21.646976 3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:21.647081 3953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:21.647945 3945 server_base.cc:1034] running on GCE node
I20260926 21:30:21.648830 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:21.650307 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:21.651561 3945 hybrid_clock.cc:648] HybridClock initialized: now 1790458221651518 us; error 67 us; skew 500 ppm
I20260926 21:30:21.651924 3945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:21 dist-test-slave-87bt krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:21.661430 3945 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:30:21.663123 3945 webserver.cc:460] Webserver started at http://127.0.97.62:35595/ using document root <none> and password file <none>
I20260926 21:30:21.663839 3945 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:21.663983 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:21.664306 3945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:21.667347 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "a211036e3c9f4eee99fe9a07e3da1a5b"
format_stamp: "Formatted at 2026-09-26 21:30:21 on dist-test-slave-87bt"
server_key: "049d1de73fc743fbda70c04f915a9965"
server_key_iv: "7e78bdacd49d5374f10b82ed1c28396e"
server_key_version: "encryptionkey@0"
I20260926 21:30:21.668125 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "a211036e3c9f4eee99fe9a07e3da1a5b"
format_stamp: "Formatted at 2026-09-26 21:30:21 on dist-test-slave-87bt"
server_key: "049d1de73fc743fbda70c04f915a9965"
server_key_iv: "7e78bdacd49d5374f10b82ed1c28396e"
server_key_version: "encryptionkey@0"
I20260926 21:30:21.673254 3945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:30:21.676648 3960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:21.678014 3945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 21:30:21.678194 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "a211036e3c9f4eee99fe9a07e3da1a5b"
format_stamp: "Formatted at 2026-09-26 21:30:21 on dist-test-slave-87bt"
server_key: "049d1de73fc743fbda70c04f915a9965"
server_key_iv: "7e78bdacd49d5374f10b82ed1c28396e"
server_key_version: "encryptionkey@0"
I20260926 21:30:21.678407 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:21.690498 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:21.694280 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:21.694651 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:21.712783 3945 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45469
I20260926 21:30:21.712813 4011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45469 every 8 connection(s)
I20260926 21:30:21.714143 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:21.717727 4012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:21.722675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 3945
I20260926 21:30:21.722896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:21.723477 388 external_mini_cluster.cc:1428] Setting key 2eb737cd15ed69d1f05aea65bb70b34f
I20260926 21:30:21.729161 4012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b: Bootstrap starting.
I20260926 21:30:21.733013 4012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:21.734163 4012 log.cc:824] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:21.737174 4012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b: No bootstrap required, opened a new log
Sep 26 21:30:21 dist-test-slave-87bt krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:21.747001 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:21.725374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.0.97.62:45469)
0926 21:30:21.726231 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:21.726247 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:21.726308 (+ 61us) server_negotiation.cc:407] Connection header received
0926 21:30:21.728648 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:21.728676 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:21.729187 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:21.729636 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:21.730961 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:21.732943 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:21.734298 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:21.735256 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:21.735411 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:21.739624 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:21.739635 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:21.739653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:21.739688 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:21.742822 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:21.743652 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:21.743658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:21.743664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:21.743838 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:21.744245 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:21.744248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:21.744250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:21.744797 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:21.745018 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:21.745470 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 21:30:21.745865 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":365,"threads_started":1}
I20260926 21:30:21.746776 4012 raft_consensus.cc:348] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211036e3c9f4eee99fe9a07e3da1a5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } }
I20260926 21:30:21.747255 4012 raft_consensus.cc:374] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:21.747373 4012 raft_consensus.cc:729] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a211036e3c9f4eee99fe9a07e3da1a5b, State: Initialized, Role: FOLLOWER
I20260926 21:30:21.747941 4012 consensus_queue.cc:260] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211036e3c9f4eee99fe9a07e3da1a5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } }
I20260926 21:30:21.748196 4012 raft_consensus.cc:388] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:21.748342 4012 raft_consensus.cc:482] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:21.748526 4012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:21.751497 4012 raft_consensus.cc:504] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211036e3c9f4eee99fe9a07e3da1a5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } }
I20260926 21:30:21.751997 4012 leader_election.cc:302] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a211036e3c9f4eee99fe9a07e3da1a5b; no voters:
I20260926 21:30:21.752529 4012 leader_election.cc:288] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:21.752660 4017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:21.753315 4017 raft_consensus.cc:686] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [term 1 LEADER]: Becoming Leader. State: Replica: a211036e3c9f4eee99fe9a07e3da1a5b, State: Running, Role: LEADER
I20260926 21:30:21.753830 4017 consensus_queue.cc:237] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211036e3c9f4eee99fe9a07e3da1a5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } }
I20260926 21:30:21.754549 4012 sys_catalog.cc:564] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:21.756590 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a211036e3c9f4eee99fe9a07e3da1a5b. Latest consensus state: current_term: 1 leader_uuid: "a211036e3c9f4eee99fe9a07e3da1a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211036e3c9f4eee99fe9a07e3da1a5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } }
I20260926 21:30:21.757090 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:21.756702 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a211036e3c9f4eee99fe9a07e3da1a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211036e3c9f4eee99fe9a07e3da1a5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45469 } } }
I20260926 21:30:21.757407 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:21.762912 4023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:21.767347 4023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:21.776075 4023 catalog_manager.cc:1347] Generated new cluster ID: 5e452a3666434b4c8a903c9d449b2ed3
I20260926 21:30:21.776227 4023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:21.790606 4023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:21.792477 4023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:21.809834 4023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a211036e3c9f4eee99fe9a07e3da1a5b: Generated new TSK 0
I20260926 21:30:21.811030 4023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:30:21.861881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45469
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:22.184685 4038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:22.185259 4038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:22.185456 4038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:22.193004 4038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:22.193163 4038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:22.193359 4038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:22.203533 4038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:22.205022 4038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:22.206362 4038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:22.219750 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:22.219801 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:22.219717 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:22.220582 4038 server_base.cc:1034] running on GCE node
I20260926 21:30:22.221184 4038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:22.222127 4038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:22.223387 4038 hybrid_clock.cc:648] HybridClock initialized: now 1790458222223341 us; error 41 us; skew 500 ppm
I20260926 21:30:22.223707 4038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:22 dist-test-slave-87bt krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:22.226735 4038 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:30:22.228156 4038 webserver.cc:460] Webserver started at http://127.0.97.1:39781/ using document root <none> and password file <none>
I20260926 21:30:22.228857 4038 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:22.228999 4038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:22.229313 4038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:22.232227 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "368e6ecff2cb463d923a2b9ea9fe12e1"
format_stamp: "Formatted at 2026-09-26 21:30:22 on dist-test-slave-87bt"
server_key: "ebd6dc91de758fc90c9c53dfaade23dd"
server_key_iv: "92b07870ce02a431ae0e86aa29f87424"
server_key_version: "encryptionkey@0"
I20260926 21:30:22.233009 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "368e6ecff2cb463d923a2b9ea9fe12e1"
format_stamp: "Formatted at 2026-09-26 21:30:22 on dist-test-slave-87bt"
server_key: "ebd6dc91de758fc90c9c53dfaade23dd"
server_key_iv: "92b07870ce02a431ae0e86aa29f87424"
server_key_version: "encryptionkey@0"
I20260926 21:30:22.237968 4038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 21:30:22.241457 4053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:22.242985 4038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:22.243175 4038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "368e6ecff2cb463d923a2b9ea9fe12e1"
format_stamp: "Formatted at 2026-09-26 21:30:22 on dist-test-slave-87bt"
server_key: "ebd6dc91de758fc90c9c53dfaade23dd"
server_key_iv: "92b07870ce02a431ae0e86aa29f87424"
server_key_version: "encryptionkey@0"
I20260926 21:30:22.243397 4038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:22.264267 4038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:22.268288 4038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:22.268570 4038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:22.269553 4038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:22.270988 4038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:22.271088 4038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:22.271276 4038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:22.271349 4038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:22.304432 4038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45673
I20260926 21:30:22.304522 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45673 every 8 connection(s)
I20260926 21:30:22.305765 4038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:22.312309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 4038
I20260926 21:30:22.312541 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:22.313174 388 external_mini_cluster.cc:1428] Setting key c1fcf6bbf45fa5e326b679f580f409f7
Sep 26 21:30:22 dist-test-slave-87bt krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:22.329397 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:22.308460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55523 (local address 127.0.97.62:45469)
0926 21:30:22.308937 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:22.308941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:22.310049 (+ 1108us) server_negotiation.cc:407] Connection header received
0926 21:30:22.311898 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:22.311907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:22.311998 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:22.312135 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:22.313895 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:22.314914 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:22.317448 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:22.317891 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:22.317992 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:22.322296 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:22.322306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:22.322313 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:22.322358 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:22.325941 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:22.326800 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:22.326809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:22.326815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:22.326967 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:22.327384 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:22.327392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:22.327398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:22.327788 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:22.327944 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:22.328653 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 21:30:22.328913 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":274,"threads_started":1}
I20260926 21:30:22.330065 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:22.309041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45469 (local address 127.0.97.1:55523)
0926 21:30:22.309834 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:22.309880 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:22.311609 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:22.312352 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:22.312361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:22.313019 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:22.313661 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:22.313704 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:22.315088 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:22.315094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:22.317274 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:22.317290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:22.318120 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:22.318129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:22.318504 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:22.319242 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:22.319281 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:22.322054 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:22.326132 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:22.326139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:22.326153 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:22.326674 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:22.327078 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:22.327081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:22.327083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:22.327265 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:22.327908 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:22.327914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:22.328249 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 21:30:22.328473 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 21:30:22.328872 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":327,"threads_started":1}
I20260926 21:30:22.331732 4166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45469
I20260926 21:30:22.332080 4166 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:22.333005 4166 heartbeater.cc:511] Master 127.0.97.62:45469 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:30:22.335511 3977 ts_manager.cc:194] Registered new tserver with Master: 368e6ecff2cb463d923a2b9ea9fe12e1 (127.0.97.1:45673)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:30:22.337129 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:30:22.361780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45469
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:22.698213 4172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:22.698570 4172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:22.698750 4172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:22.705988 4172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:22.706154 4172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:22.706336 4172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:22.716161 4172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:22.717382 4172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:22.718667 4172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:22.731709 4178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:22.731729 4177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:22.732314 4172 server_base.cc:1034] running on GCE node
W20260926 21:30:22.732513 4180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:22.733192 4172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:22.734130 4172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:22.735323 4172 hybrid_clock.cc:648] HybridClock initialized: now 1790458222735251 us; error 67 us; skew 500 ppm
I20260926 21:30:22.735633 4172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:22 dist-test-slave-87bt krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:22.738826 4172 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:30:22.740175 4172 webserver.cc:460] Webserver started at http://127.0.97.2:37191/ using document root <none> and password file <none>
I20260926 21:30:22.740845 4172 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:22.740968 4172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:22.741285 4172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:22.744127 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "a9310fde861b41ca91fd67d777b0e0ee"
format_stamp: "Formatted at 2026-09-26 21:30:22 on dist-test-slave-87bt"
server_key: "44be7931a58a07cd532be633093f5a68"
server_key_iv: "9c29f2e4cc14997fc6c9254a4157adbb"
server_key_version: "encryptionkey@0"
I20260926 21:30:22.744911 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a9310fde861b41ca91fd67d777b0e0ee"
format_stamp: "Formatted at 2026-09-26 21:30:22 on dist-test-slave-87bt"
server_key: "44be7931a58a07cd532be633093f5a68"
server_key_iv: "9c29f2e4cc14997fc6c9254a4157adbb"
server_key_version: "encryptionkey@0"
I20260926 21:30:22.750098 4172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:30:22.753538 4187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:22.754962 4172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:22.755133 4172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "a9310fde861b41ca91fd67d777b0e0ee"
format_stamp: "Formatted at 2026-09-26 21:30:22 on dist-test-slave-87bt"
server_key: "44be7931a58a07cd532be633093f5a68"
server_key_iv: "9c29f2e4cc14997fc6c9254a4157adbb"
server_key_version: "encryptionkey@0"
I20260926 21:30:22.755348 4172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:22.776336 4172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:22.780774 4172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:22.781097 4172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:22.782192 4172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:22.783924 4172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:22.784062 4172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:22.784222 4172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:22.784300 4172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:22.819885 4172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42347
I20260926 21:30:22.819973 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42347 every 8 connection(s)
I20260926 21:30:22.821164 4172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:22.824836 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 4172
I20260926 21:30:22.825062 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:22.825578 388 external_mini_cluster.cc:1428] Setting key 6e94531b8fa02de77901cc1923157042
Sep 26 21:30:22 dist-test-slave-87bt krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:22.846354 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:22.823817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36195 (local address 127.0.97.62:45469)
0926 21:30:22.824001 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:22.824006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:22.825294 (+ 1288us) server_negotiation.cc:407] Connection header received
0926 21:30:22.827397 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:22.827404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:22.827488 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:22.827572 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:22.829049 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:22.830367 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:22.833283 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:22.834127 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:22.835323 (+ 1196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:22.839304 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:22.839312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:22.839318 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:22.839370 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:22.842595 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:22.843478 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:22.843488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:22.843495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:22.843694 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:22.844213 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:22.844219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:22.844223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:22.844636 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:22.844844 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:22.845625 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 21:30:22.845908 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:30:22.847222 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:22.824301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45469 (local address 127.0.97.2:36195)
0926 21:30:22.825122 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:22.825168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:22.827103 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:22.827836 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:22.827844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:22.828344 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:22.828882 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:22.828899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:22.830628 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:22.830638 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:22.833076 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:22.833096 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:22.834449 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:22.834460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:22.834815 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:22.835748 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:22.835789 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:22.838986 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:22.842770 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:22.842781 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:22.842800 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:22.843320 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:22.843854 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:22.843859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:22.843865 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:22.844083 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:22.844804 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:22.844810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:22.845224 (+ 414us) client_negotiation.cc:769] Sending connection context
0926 21:30:22.845485 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 21:30:22.845888 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":361,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:30:22.848883 4300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45469
I20260926 21:30:22.849169 4300 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:22.849951 4300 heartbeater.cc:511] Master 127.0.97.62:45469 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 21:30:22.851641 3977 ts_manager.cc:194] Registered new tserver with Master: a9310fde861b41ca91fd67d777b0e0ee (127.0.97.2:42347)
I20260926 21:30:22.852540 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:30:22.875753 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45469
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:23.196882 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:23.197211 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:23.197381 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:23.204747 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:23.204867 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:23.205062 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:23.215629 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:23.216929 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:23.218227 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:23.231933 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:23.231966 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:23.231896 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:23.232693 4305 server_base.cc:1034] running on GCE node
I20260926 21:30:23.233307 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:23.234230 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:23.235455 4305 hybrid_clock.cc:648] HybridClock initialized: now 1790458223235383 us; error 67 us; skew 500 ppm
I20260926 21:30:23.235769 4305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:23.238806 4305 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:30:23.240150 4305 webserver.cc:460] Webserver started at http://127.0.97.3:41673/ using document root <none> and password file <none>
I20260926 21:30:23.240805 4305 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:23.240952 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:23.241256 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:23.244071 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "a7af6811f22848f78fe154eb6a97a882"
format_stamp: "Formatted at 2026-09-26 21:30:23 on dist-test-slave-87bt"
server_key: "b2249ae62c18b89658c5369b1f30f27e"
server_key_iv: "25ba880b56f4627147159c4537962009"
server_key_version: "encryptionkey@0"
I20260926 21:30:23.244952 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a7af6811f22848f78fe154eb6a97a882"
format_stamp: "Formatted at 2026-09-26 21:30:23 on dist-test-slave-87bt"
server_key: "b2249ae62c18b89658c5369b1f30f27e"
server_key_iv: "25ba880b56f4627147159c4537962009"
server_key_version: "encryptionkey@0"
I20260926 21:30:23.250020 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:30:23.253567 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:23.254992 4305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:23.255160 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "a7af6811f22848f78fe154eb6a97a882"
format_stamp: "Formatted at 2026-09-26 21:30:23 on dist-test-slave-87bt"
server_key: "b2249ae62c18b89658c5369b1f30f27e"
server_key_iv: "25ba880b56f4627147159c4537962009"
server_key_version: "encryptionkey@0"
I20260926 21:30:23.255354 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:23.278620 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:23.282843 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:23.283082 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:23.283979 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:23.285180 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:23.285250 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:23.285329 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:23.285374 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:23.319398 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34213
I20260926 21:30:23.319454 4432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34213 every 8 connection(s)
I20260926 21:30:23.320645 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:23.324251 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 4305
I20260926 21:30:23.324437 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:23.324954 388 external_mini_cluster.cc:1428] Setting key 980eb0cc063292bc72ef1cb1351ad854
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:23.340363 4166 heartbeater.cc:503] Master 127.0.97.62:45469 was elected leader, sending a full tablet report...
I20260926 21:30:23.341576 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:23.323126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47433 (local address 127.0.97.62:45469)
0926 21:30:23.323331 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:23.323337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:23.324719 (+ 1382us) server_negotiation.cc:407] Connection header received
0926 21:30:23.326943 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:23.326948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:23.327036 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:23.327140 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:23.328547 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.329281 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.330874 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.331529 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.331689 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:23.335247 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:23.335252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:23.335256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:23.335300 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:23.337905 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.338828 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.338833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.338837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.339060 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.339564 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.339569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.339573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.339918 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:23.340120 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:23.340982 (+ 862us) server_negotiation.cc:299] Negotiation successful
0926 21:30:23.341252 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:30:23.342525 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:23.323673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45469 (local address 127.0.97.3:47433)
0926 21:30:23.324551 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:23.324609 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:23.326682 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:23.327366 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:23.327374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:23.327858 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:23.328391 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:23.328406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.329430 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.329437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:23.330720 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:23.330731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.331771 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.331781 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:23.332141 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:23.332805 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:23.332827 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:23.334998 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:23.338051 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:23.338059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:23.338076 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:23.338694 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:23.339184 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:23.339188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:23.339191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:23.339430 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:23.340065 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:23.340074 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:23.340492 (+ 418us) client_negotiation.cc:769] Sending connection context
0926 21:30:23.340814 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 21:30:23.341317 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":353,"threads_started":1}
I20260926 21:30:23.343772 4433 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45469
I20260926 21:30:23.344055 4433 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:23.344678 4433 heartbeater.cc:511] Master 127.0.97.62:45469 requested a full tablet report, sending...
I20260926 21:30:23.345888 3977 ts_manager.cc:194] Registered new tserver with Master: a7af6811f22848f78fe154eb6a97a882 (127.0.97.3:34213)
I20260926 21:30:23.346714 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47433
I20260926 21:30:23.352344 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:23.398680 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:23.384131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.0.97.1:45673)
0926 21:30:23.384779 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:23.384790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:23.384822 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:30:23.384897 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:23.384904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:23.385117 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:23.385297 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:23.386691 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.387681 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.388890 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.389450 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.389582 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:23.393426 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:23.393435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:23.393449 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:23.393477 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:23.396030 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.396633 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.396643 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.396648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.396765 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.397130 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.397135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.397139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.397442 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:23.397734 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:23.398043 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 21:30:23.398302 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":311,"threads_started":1}
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:23.415175 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:23.402522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.0.97.62:45469)
0926 21:30:23.402749 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:23.402756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:23.402989 (+ 233us) server_negotiation.cc:407] Connection header received
0926 21:30:23.403185 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:23.403191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:23.403269 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:23.403371 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:23.404500 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.405185 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.406394 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.406857 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.406983 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:23.410595 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:23.410600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:23.410603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:23.410633 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:23.412967 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.413491 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.413494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.413496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.413607 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.413903 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.413906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.413908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.414206 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:23.414371 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:23.414685 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 21:30:23.414838 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 21:30:23.415719 3977 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50866
I20260926 21:30:23.416637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 4038
I20260926 21:30:23.428470 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:23.417503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.0.97.1:45673)
0926 21:30:23.417725 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:23.417731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:23.417869 (+ 138us) server_negotiation.cc:407] Connection header received
0926 21:30:23.417972 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:23.417976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:23.418057 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:23.418147 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:23.419255 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.419893 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.420996 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.421407 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.421495 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:23.423658 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:23.423662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:23.423665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:23.423693 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:23.426271 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.426875 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.426879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.426881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.426985 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.427349 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.427352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.427354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.427572 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:23.427741 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:23.427994 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 21:30:23.428154 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 21:30:23.429076 4161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:23.510645 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 4038
I20260926 21:30:23.527365 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 4172
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:30:23.543767 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:23.528294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.0.97.2:42347)
0926 21:30:23.528788 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:23.528795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:23.528820 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:30:23.528875 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:23.528882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:23.529321 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:23.529548 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:23.530864 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.531875 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.533129 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.533686 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.533825 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:23.538198 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:23.538209 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:23.538224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:23.538266 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:23.540959 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.541598 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.541605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.541608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.541729 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.542126 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.542130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.542132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.542542 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:23.542856 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:23.543152 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 21:30:23.543411 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":244,"threads_started":1}
I20260926 21:30:23.545039 4295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:23.626602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 4172
I20260926 21:30:23.642387 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 4305
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:30:23.658605 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:23.643432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.0.97.3:34213)
0926 21:30:23.644054 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:23.644061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:23.644088 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:30:23.644145 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:23.644152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:23.644353 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:23.644540 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:23.645789 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.646858 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.648135 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:23.648691 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:23.648829 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:23.652976 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:23.652988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:23.653001 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:23.653038 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:23.655883 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.656520 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.656533 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.656535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.656649 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:23.657011 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:23.657018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:23.657021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:23.657353 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:23.657674 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:23.657947 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 21:30:23.658207 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":323,"threads_started":1}
I20260926 21:30:23.659912 4428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:23.742146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 4305
I20260926 21:30:23.757417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 3945
I20260926 21:30:23.758630 4007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:23.859485 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 3945
2026-09-26T21:30:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3924 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[4452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[4452](info): set up 2 sockets
Sep 26 21:30:23 dist-test-slave-87bt krb5kdc[4452](info): commencing operation
krb5kdc: starting...
W20260926 21:30:24.955327 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:25 dist-test-slave-87bt krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:25.150516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43209
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:25.465619 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:25.465919 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:25.466077 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:25.473410 4465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:25.473519 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:25.473628 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:25.473729 4465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:25.473825 4465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:25.483698 4465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:25.484952 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:25.486296 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:25.497617 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:25.497695 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:25.497673 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:25.498648 4465 server_base.cc:1034] running on GCE node
I20260926 21:30:25.499439 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:25.500877 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:25.502075 4465 hybrid_clock.cc:648] HybridClock initialized: now 1790458225502020 us; error 56 us; skew 500 ppm
I20260926 21:30:25.502473 4465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:25 dist-test-slave-87bt krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:25.505527 4465 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:30:25.506659 4465 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:25.506748 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:25.506987 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:25.509744 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "5fa35ea571cd4b2886a7726c1ebc4811"
format_stamp: "Formatted at 2026-09-26 21:30:25 on dist-test-slave-87bt"
server_key: "1b811937029c10780c656752c9919405"
server_key_iv: "5cff74026554c72f4ea320c84d0995ed"
server_key_version: "encryptionkey@0"
I20260926 21:30:25.510506 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "5fa35ea571cd4b2886a7726c1ebc4811"
format_stamp: "Formatted at 2026-09-26 21:30:25 on dist-test-slave-87bt"
server_key: "1b811937029c10780c656752c9919405"
server_key_iv: "5cff74026554c72f4ea320c84d0995ed"
server_key_version: "encryptionkey@0"
I20260926 21:30:25.515436 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 21:30:25.518832 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:25.520089 4465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:30:25.520264 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "5fa35ea571cd4b2886a7726c1ebc4811"
format_stamp: "Formatted at 2026-09-26 21:30:25 on dist-test-slave-87bt"
server_key: "1b811937029c10780c656752c9919405"
server_key_iv: "5cff74026554c72f4ea320c84d0995ed"
server_key_version: "encryptionkey@0"
I20260926 21:30:25.520495 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:25.543601 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:25.548050 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:25.548323 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:25.566174 4465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43209
I20260926 21:30:25.566195 4530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43209 every 8 connection(s)
I20260926 21:30:25.567066 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:25.567708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 4465
I20260926 21:30:25.567939 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:25.568560 388 external_mini_cluster.cc:1428] Setting key 31ab331d28b63a52264f4d78e3bbbe2f
I20260926 21:30:25.571640 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:30:25 dist-test-slave-87bt krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:25.584342 4531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811: Bootstrap starting.
I20260926 21:30:25.587361 4531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:25.588351 4531 log.cc:824] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:25.591005 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:25.570674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.0.97.62:43209)
0926 21:30:25.571462 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:25.571473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:25.571512 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:30:25.573124 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:25.573144 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:25.573503 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:25.573816 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:25.575729 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:25.577028 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:25.578858 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:25.579479 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:25.579601 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:25.584055 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:25.584061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:25.584074 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:25.584110 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:25.587062 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:25.587774 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:25.587780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:25.587787 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:25.587940 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:25.588309 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:25.588319 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:25.588320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:25.588898 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:25.589204 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:25.589642 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 21:30:25.590023 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":364,"threads_started":1}
I20260926 21:30:25.591246 4531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811: No bootstrap required, opened a new log
I20260926 21:30:25.598961 4531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa35ea571cd4b2886a7726c1ebc4811" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43209 } }
I20260926 21:30:25.599363 4531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:25.599475 4531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa35ea571cd4b2886a7726c1ebc4811, State: Initialized, Role: FOLLOWER
I20260926 21:30:25.600106 4531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa35ea571cd4b2886a7726c1ebc4811" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43209 } }
I20260926 21:30:25.600386 4531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:25.600518 4531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:25.600639 4531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:25.602803 4531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa35ea571cd4b2886a7726c1ebc4811" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43209 } }
I20260926 21:30:25.603317 4531 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa35ea571cd4b2886a7726c1ebc4811; no voters:
I20260926 21:30:25.603888 4531 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:25.604024 4536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:25.605165 4536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa35ea571cd4b2886a7726c1ebc4811, State: Running, Role: LEADER
I20260926 21:30:25.605748 4536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa35ea571cd4b2886a7726c1ebc4811" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43209 } }
I20260926 21:30:25.606285 4531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:25.608609 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa35ea571cd4b2886a7726c1ebc4811. Latest consensus state: current_term: 1 leader_uuid: "5fa35ea571cd4b2886a7726c1ebc4811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa35ea571cd4b2886a7726c1ebc4811" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43209 } } }
I20260926 21:30:25.608979 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:25.609557 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa35ea571cd4b2886a7726c1ebc4811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa35ea571cd4b2886a7726c1ebc4811" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43209 } } }
I20260926 21:30:25.609802 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:25.612169 4545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:25.617071 4545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:25.625994 4545 catalog_manager.cc:1347] Generated new cluster ID: 6c6d650567f24c50b9589d8bcb0ef3fa
I20260926 21:30:25.626108 4545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:25.648123 4545 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:25.649919 4545 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:25.660825 4545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811: Generated new TSK 0
I20260926 21:30:25.661909 4545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:30:25.718091 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43209
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:26.042395 4557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:26.042775 4557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:26.042953 4557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:26.050228 4557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:26.050384 4557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:26.050592 4557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:26.060328 4557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:26.061688 4557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:26.063057 4557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:26.075848 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:26.075763 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:26.075798 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:26.076637 4557 server_base.cc:1034] running on GCE node
I20260926 21:30:26.077154 4557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:26.078018 4557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:26.079218 4557 hybrid_clock.cc:648] HybridClock initialized: now 1790458226079183 us; error 34 us; skew 500 ppm
I20260926 21:30:26.079500 4557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:26 dist-test-slave-87bt krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:26.082564 4557 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:30:26.083817 4557 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:26.083911 4557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:26.084152 4557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:26.087028 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "92585ac4b07d41508d35cd653981fd9a"
format_stamp: "Formatted at 2026-09-26 21:30:26 on dist-test-slave-87bt"
server_key: "69462ea73a3c4cf71aefe54b3924d1ff"
server_key_iv: "046cd836a82e954173e8136869e14cd0"
server_key_version: "encryptionkey@0"
I20260926 21:30:26.087786 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "92585ac4b07d41508d35cd653981fd9a"
format_stamp: "Formatted at 2026-09-26 21:30:26 on dist-test-slave-87bt"
server_key: "69462ea73a3c4cf71aefe54b3924d1ff"
server_key_iv: "046cd836a82e954173e8136869e14cd0"
server_key_version: "encryptionkey@0"
I20260926 21:30:26.092788 4557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:30:26.096297 4571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:26.097661 4557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:26.097841 4557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "92585ac4b07d41508d35cd653981fd9a"
format_stamp: "Formatted at 2026-09-26 21:30:26 on dist-test-slave-87bt"
server_key: "69462ea73a3c4cf71aefe54b3924d1ff"
server_key_iv: "046cd836a82e954173e8136869e14cd0"
server_key_version: "encryptionkey@0"
I20260926 21:30:26.098057 4557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:26.121443 4557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:26.126000 4557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:26.126283 4557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:26.127352 4557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:26.128616 4557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:26.128741 4557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:26.128871 4557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:26.128968 4557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:26.161536 4557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34329
I20260926 21:30:26.161588 4683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34329 every 8 connection(s)
I20260926 21:30:26.162417 4557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:26.168150 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 4557
I20260926 21:30:26.168416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:26.169191 388 external_mini_cluster.cc:1428] Setting key 436c048d101666dd30c5cf61130efbd5
Sep 26 21:30:26 dist-test-slave-87bt krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:26.184065 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:26.164935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55763 (local address 127.0.97.62:43209)
0926 21:30:26.165395 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:26.165400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:26.166240 (+ 840us) server_negotiation.cc:407] Connection header received
0926 21:30:26.168137 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:26.168142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:26.168243 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:26.168374 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:26.170373 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:26.171220 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:26.172843 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:26.173229 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:26.173366 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:26.176981 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:26.176990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:26.176998 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:26.177046 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:26.180649 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:26.181447 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:26.181455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:26.181462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:26.181612 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:26.182027 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:26.182035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:26.182040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:26.182457 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:26.182606 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:26.183312 (+ 706us) server_negotiation.cc:299] Negotiation successful
0926 21:30:26.183573 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":241,"threads_started":1}
I20260926 21:30:26.184711 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:26.165387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43209 (local address 127.0.97.1:55763)
0926 21:30:26.166054 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:26.166098 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:26.167860 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:26.168660 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:26.168675 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:26.169481 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:26.170185 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:26.170204 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:26.171399 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:26.171403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:26.172656 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:26.172669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:26.173468 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:26.173475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:26.173707 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:26.174476 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:26.174514 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:26.176718 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:26.180835 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:26.180842 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:26.180858 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:26.181309 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:26.181727 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:26.181730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:26.181732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:26.181911 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:26.182586 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:26.182592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:26.182935 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 21:30:26.183163 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 21:30:26.183566 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":259,"threads_started":1}
I20260926 21:30:26.186373 4684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43209
I20260926 21:30:26.186734 4684 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:26.187381 4684 heartbeater.cc:511] Master 127.0.97.62:43209 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:30:26.189504 4496 ts_manager.cc:194] Registered new tserver with Master: 92585ac4b07d41508d35cd653981fd9a (127.0.97.1:34329)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:30:26.191249 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:30:26.214880 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43209
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:26.532184 4690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:26.532487 4690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:26.532653 4690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:26.540373 4690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:26.540549 4690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:26.540732 4690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:26.551093 4690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:26.552484 4690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:26.553810 4690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:26.567067 4696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:26.567253 4698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:26.567607 4695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:26.569700 4690 server_base.cc:1034] running on GCE node
I20260926 21:30:26.570318 4690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:26.571343 4690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:26.572571 4690 hybrid_clock.cc:648] HybridClock initialized: now 1790458226572517 us; error 48 us; skew 500 ppm
I20260926 21:30:26.572922 4690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:26 dist-test-slave-87bt krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:26.576731 4690 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:30:26.578008 4690 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:26.578140 4690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:26.578513 4690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:26.581480 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "75421089c20f4289ba38750d161ebcd0"
format_stamp: "Formatted at 2026-09-26 21:30:26 on dist-test-slave-87bt"
server_key: "2cc14cca177f847d5115232524dc1b2a"
server_key_iv: "fce901e83969bdfe12c04698fd971e7f"
server_key_version: "encryptionkey@0"
I20260926 21:30:26.582571 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "75421089c20f4289ba38750d161ebcd0"
format_stamp: "Formatted at 2026-09-26 21:30:26 on dist-test-slave-87bt"
server_key: "2cc14cca177f847d5115232524dc1b2a"
server_key_iv: "fce901e83969bdfe12c04698fd971e7f"
server_key_version: "encryptionkey@0"
I20260926 21:30:26.587832 4690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:30:26.591949 4704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:26.593581 4690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:30:26.593758 4690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "75421089c20f4289ba38750d161ebcd0"
format_stamp: "Formatted at 2026-09-26 21:30:26 on dist-test-slave-87bt"
server_key: "2cc14cca177f847d5115232524dc1b2a"
server_key_iv: "fce901e83969bdfe12c04698fd971e7f"
server_key_version: "encryptionkey@0"
I20260926 21:30:26.593984 4690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:26.610718 4690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:26.614740 4690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:26.615020 4690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:26.616036 4690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:26.617355 4690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:26.617477 4690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:26.617602 4690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:26.617686 4690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:26.649981 4690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43325
I20260926 21:30:26.650027 4816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43325 every 8 connection(s)
I20260926 21:30:26.650831 4690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:26.653443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 4690
I20260926 21:30:26.653626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:26.654055 388 external_mini_cluster.cc:1428] Setting key 06eb66e03d55ae577b3f090f0ef63100
Sep 26 21:30:26 dist-test-slave-87bt krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 21:30:26.676126 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:26.653223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56307 (local address 127.0.97.62:43209)
0926 21:30:26.653452 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:26.653458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:26.654751 (+ 1293us) server_negotiation.cc:407] Connection header received
0926 21:30:26.656898 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:26.656902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:26.656962 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:26.657042 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:26.659230 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:26.659935 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:26.662294 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:26.663052 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:26.663192 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:26.670677 (+ 7485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:26.670685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:26.670690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:26.670730 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:26.673160 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:26.673956 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:26.673960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:26.673962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:26.674073 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:26.674565 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:26.674569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:26.674571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:26.674825 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:26.674930 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:26.675618 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 21:30:26.675803 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 21:30:26.677392 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:26.653761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43209 (local address 127.0.97.2:56307)
0926 21:30:26.654572 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:26.654623 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:26.656593 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:26.657325 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:26.657340 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:26.658150 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:26.659032 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:26.659067 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:26.660080 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:26.660088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:26.662147 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:26.662159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:26.663252 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:26.663257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:26.663462 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:26.667536 (+ 4074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:26.667560 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:26.670339 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:26.673352 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:26.673362 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:26.673385 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:26.673845 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:26.674165 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:26.674168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:26.674170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:26.674415 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:26.674931 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:26.674937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:26.675250 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 21:30:26.675529 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 21:30:26.675991 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":333,"threads_started":1}
I20260926 21:30:26.678915 4817 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43209
I20260926 21:30:26.679222 4817 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:26.679872 4817 heartbeater.cc:511] Master 127.0.97.62:43209 requested a full tablet report, sending...
I20260926 21:30:26.681391 4496 ts_manager.cc:194] Registered new tserver with Master: 75421089c20f4289ba38750d161ebcd0 (127.0.97.2:43325)
I20260926 21:30:26.682385 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:30:26.701121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43209
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:27.024734 4822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:27.025117 4822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:27.025298 4822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:27.032779 4822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:27.032919 4822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:27.033125 4822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:27.043412 4822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:27.044842 4822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:27.046272 4822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:27.060401 4827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:27.060488 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:27.060447 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:27.061291 4822 server_base.cc:1034] running on GCE node
I20260926 21:30:27.061928 4822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:27.062964 4822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:27.064162 4822 hybrid_clock.cc:648] HybridClock initialized: now 1790458227064121 us; error 30 us; skew 500 ppm
I20260926 21:30:27.064591 4822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:27 dist-test-slave-87bt krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:27.067703 4822 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:30:27.068897 4822 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:27.069033 4822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:27.069375 4822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:27.072482 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "d9ae359b730d49f3afd42674e63ab859"
format_stamp: "Formatted at 2026-09-26 21:30:27 on dist-test-slave-87bt"
server_key: "efc4e2e64bf18e931ae9b6a7d27a3727"
server_key_iv: "2c157dd2e3694408ba0387b04eb1eb13"
server_key_version: "encryptionkey@0"
I20260926 21:30:27.073331 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d9ae359b730d49f3afd42674e63ab859"
format_stamp: "Formatted at 2026-09-26 21:30:27 on dist-test-slave-87bt"
server_key: "efc4e2e64bf18e931ae9b6a7d27a3727"
server_key_iv: "2c157dd2e3694408ba0387b04eb1eb13"
server_key_version: "encryptionkey@0"
I20260926 21:30:27.078374 4822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:30:27.081746 4836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:27.083129 4822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:27.083307 4822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "d9ae359b730d49f3afd42674e63ab859"
format_stamp: "Formatted at 2026-09-26 21:30:27 on dist-test-slave-87bt"
server_key: "efc4e2e64bf18e931ae9b6a7d27a3727"
server_key_iv: "2c157dd2e3694408ba0387b04eb1eb13"
server_key_version: "encryptionkey@0"
I20260926 21:30:27.083520 4822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:27.108572 4822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:27.112771 4822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:27.113045 4822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:27.114027 4822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:27.115371 4822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:27.115495 4822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:27.115619 4822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:27.115705 4822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:27.148041 4822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43661
I20260926 21:30:27.148087 4948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43661 every 8 connection(s)
I20260926 21:30:27.148877 4822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:27.149838 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 4822
I20260926 21:30:27.150023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:27.150537 388 external_mini_cluster.cc:1428] Setting key c5eec8cc61dba4b930c39c8df8501d0d
Sep 26 21:30:27 dist-test-slave-87bt krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:27.168874 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.151703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44151 (local address 127.0.97.62:43209)
0926 21:30:27.151890 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.151898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.153105 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 21:30:27.155052 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.155057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.155135 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.155247 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:27.156659 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.157414 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.158977 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.159363 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.159489 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.163075 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:27.163080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:27.163083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:27.163112 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:27.166283 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.167059 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.167063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.167065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.167175 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.167527 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.167530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.167532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.167759 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:27.167869 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.168470 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.168626 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:30:27.169766 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.152177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43209 (local address 127.0.97.3:44151)
0926 21:30:27.152958 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:27.152999 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:27.154813 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:27.155460 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:27.155467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:27.155959 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:27.156494 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.156509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.157565 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.157570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.158840 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.158851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.159583 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.159590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.159819 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.160412 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:27.160433 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:27.162896 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:27.166414 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:27.166444 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:27.166459 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:27.166952 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:27.167275 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:27.167278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:27.167280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:27.167429 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:27.167867 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:27.167872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:27.168153 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 21:30:27.168370 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 21:30:27.168728 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":354,"threads_started":1}
I20260926 21:30:27.171057 4949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43209
I20260926 21:30:27.171350 4949 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:27.171979 4949 heartbeater.cc:511] Master 127.0.97.62:43209 requested a full tablet report, sending...
I20260926 21:30:27.173355 4496 ts_manager.cc:194] Registered new tserver with Master: d9ae359b730d49f3afd42674e63ab859 (127.0.97.3:43661)
I20260926 21:30:27.174165 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44151
I20260926 21:30:27.177927 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:30:27.194350 4684 heartbeater.cc:503] Master 127.0.97.62:43209 was elected leader, sending a full tablet report...
I20260926 21:30:27.194506 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.182596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.0.97.62:43209)
0926 21:30:27.182769 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.182774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.183241 (+ 467us) server_negotiation.cc:407] Connection header received
0926 21:30:27.183386 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.183389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.183448 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.183533 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:27.184917 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.185606 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.186899 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.187413 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.187510 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.189736 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:27.189741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:27.189744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:27.189784 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:27.192171 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.192812 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.192816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.192818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.192929 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.193238 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.193241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.193243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.193518 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:27.193685 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.193982 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.194168 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:30:27.198985 4495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:27.205703 4495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:27.227865 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.217858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.0.97.1:34329)
0926 21:30:27.218955 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.218965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.218992 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:30:27.219101 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.219112 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.219311 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.219508 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:27.220211 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.221785 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.224751 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.226274 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.226387 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.226689 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.227371 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.227537 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":789,"threads_started":1}
I20260926 21:30:27.228823 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.217731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34329 (local address 127.0.0.1:44376)
0926 21:30:27.218512 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:27.218566 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:27.218750 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:27.219508 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:27.219515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:27.219550 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:27.220009 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.220024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.221990 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.221995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.224563 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.224577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.226878 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.226885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.227071 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.227129 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 21:30:27.228249 (+ 1120us) client_negotiation.cc:240] Negotiation successful
0926 21:30:27.228442 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":350,"threads_started":1}
I20260926 21:30:27.231417 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.218986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43661 (local address 127.0.0.1:33342)
0926 21:30:27.220151 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:27.220169 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:27.220263 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:27.222338 (+ 2075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:27.222342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:27.222366 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:27.222868 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.222881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.226144 (+ 3263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.226150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.228294 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.228307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.230623 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.230629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.230795 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.230834 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 21:30:27.230941 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 21:30:27.231091 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":886,"threads_started":1}
I20260926 21:30:27.231719 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.219393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.0.97.3:43661)
0926 21:30:27.221710 (+ 2317us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.221721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.221766 (+ 45us) server_negotiation.cc:407] Connection header received
0926 21:30:27.221859 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.221868 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.222119 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.223039 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:27.223322 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.225995 (+ 2673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.229493 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.230389 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.230564 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.230801 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.231216 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.231379 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2163,"thread_start_us":1065,"threads_started":1}
I20260926 21:30:27.235459 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 7fae9ce64c8845b4ae288a8435493b0d (DEFAULT_TABLE table=test-table [id=1843112e3d8a4f44b78847c7e144b18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:27.236034 4884 tablet_service.cc:1467] Processing CreateTablet for tablet 7fae9ce64c8845b4ae288a8435493b0d (DEFAULT_TABLE table=test-table [id=1843112e3d8a4f44b78847c7e144b18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:27.237075 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fae9ce64c8845b4ae288a8435493b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:27.237352 4884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fae9ce64c8845b4ae288a8435493b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:27.238134 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.220969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43325 (local address 127.0.0.1:36866)
0926 21:30:27.229116 (+ 8147us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:27.229135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:27.229225 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:27.232823 (+ 3598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:27.232828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:27.232855 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:27.232998 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.233008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.234740 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.234746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.235866 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.235877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.237469 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.237474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.237604 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.237626 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:30:27.237722 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 21:30:27.237859 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8061,"thread_start_us":10006,"threads_started":1}
I20260926 21:30:27.238245 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.220534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.0.97.2:43325)
0926 21:30:27.232302 (+ 11768us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.232309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.232332 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:30:27.232409 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.232414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.232580 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.232832 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:27.233143 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.234612 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.236018 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.237097 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.237244 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.237365 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.237780 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.237925 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11650,"thread_start_us":11541,"threads_started":1}
I20260926 21:30:27.241295 4752 tablet_service.cc:1467] Processing CreateTablet for tablet 7fae9ce64c8845b4ae288a8435493b0d (DEFAULT_TABLE table=test-table [id=1843112e3d8a4f44b78847c7e144b18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:27.242291 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fae9ce64c8845b4ae288a8435493b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:27.251684 4968 tablet_bootstrap.cc:492] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Bootstrap starting.
I20260926 21:30:27.252485 4967 tablet_bootstrap.cc:492] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Bootstrap starting.
I20260926 21:30:27.253558 4969 tablet_bootstrap.cc:492] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Bootstrap starting.
I20260926 21:30:27.256212 4967 tablet_bootstrap.cc:654] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:27.256969 4969 tablet_bootstrap.cc:654] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:27.257330 4967 log.cc:824] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:27.257668 4968 tablet_bootstrap.cc:654] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:27.258154 4969 log.cc:824] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:27.258905 4968 log.cc:824] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:27.260941 4969 tablet_bootstrap.cc:492] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: No bootstrap required, opened a new log
I20260926 21:30:27.261018 4967 tablet_bootstrap.cc:492] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: No bootstrap required, opened a new log
I20260926 21:30:27.261204 4969 ts_tablet_manager.cc:1397] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:30:27.261283 4967 ts_tablet_manager.cc:1397] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 21:30:27.261674 4968 tablet_bootstrap.cc:492] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: No bootstrap required, opened a new log
I20260926 21:30:27.261917 4968 ts_tablet_manager.cc:1397] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 21:30:27.268546 4969 raft_consensus.cc:348] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
I20260926 21:30:27.268916 4969 raft_consensus.cc:374] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:27.269074 4969 raft_consensus.cc:729] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75421089c20f4289ba38750d161ebcd0, State: Initialized, Role: FOLLOWER
I20260926 21:30:27.269685 4969 consensus_queue.cc:260] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
I20260926 21:30:27.270792 4817 heartbeater.cc:503] Master 127.0.97.62:43209 was elected leader, sending a full tablet report...
I20260926 21:30:27.271811 4967 raft_consensus.cc:348] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
I20260926 21:30:27.272289 4967 raft_consensus.cc:374] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:27.272425 4967 raft_consensus.cc:729] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92585ac4b07d41508d35cd653981fd9a, State: Initialized, Role: FOLLOWER
I20260926 21:30:27.272243 4968 raft_consensus.cc:348] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
I20260926 21:30:27.272693 4968 raft_consensus.cc:374] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:27.272804 4969 ts_tablet_manager.cc:1428] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260926 21:30:27.272833 4968 raft_consensus.cc:729] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ae359b730d49f3afd42674e63ab859, State: Initialized, Role: FOLLOWER
I20260926 21:30:27.273144 4967 consensus_queue.cc:260] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
I20260926 21:30:27.273476 4968 consensus_queue.cc:260] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
I20260926 21:30:27.275349 4967 ts_tablet_manager.cc:1428] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 21:30:27.276168 4949 heartbeater.cc:503] Master 127.0.97.62:43209 was elected leader, sending a full tablet report...
I20260926 21:30:27.276804 4968 ts_tablet_manager.cc:1428] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260926 21:30:27.378268 4973 raft_consensus.cc:482] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:27.378598 4973 raft_consensus.cc:504] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
I20260926 21:30:27.380296 4973 leader_election.cc:288] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92585ac4b07d41508d35cd653981fd9a (127.0.97.1:34329), d9ae359b730d49f3afd42674e63ab859 (127.0.97.3:43661)
I20260926 21:30:27.387223 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.380746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38563 (local address 127.0.97.1:34329)
0926 21:30:27.380981 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.380987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.381306 (+ 319us) server_negotiation.cc:407] Connection header received
0926 21:30:27.381527 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.381535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.381620 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.381712 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:27.382383 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.383357 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.385245 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.386158 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.386268 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.386341 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.386837 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.386970 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:30:27.387305 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.380548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34329 (local address 127.0.97.2:38563)
0926 21:30:27.381212 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:27.381231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:27.381383 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:27.381829 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:27.381833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:27.381857 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:27.382221 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.382233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.383487 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.383492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.385098 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.385110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.386377 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.386383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.386584 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.386623 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 21:30:27.386787 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 21:30:27.386939 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":321,"threads_started":1}
I20260926 21:30:27.387646 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.381511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35097 (local address 127.0.97.3:43661)
0926 21:30:27.381732 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.381738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.382044 (+ 306us) server_negotiation.cc:407] Connection header received
0926 21:30:27.382172 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.382178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.382262 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.382351 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:27.382999 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.383909 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.385695 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.386642 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.386773 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.386867 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.387293 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.387426 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:30:27.387724 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.381206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43661 (local address 127.0.97.2:35097)
0926 21:30:27.381936 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:27.381956 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:27.382053 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:27.382380 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:27.382384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:27.382408 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:27.382833 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.382845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.384040 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.384044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.385545 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:27.385557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.386907 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.386914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:27.387117 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.387146 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 21:30:27.387251 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 21:30:27.387430 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":398,"threads_started":1}
I20260926 21:30:27.388482 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fae9ce64c8845b4ae288a8435493b0d" candidate_uuid: "75421089c20f4289ba38750d161ebcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92585ac4b07d41508d35cd653981fd9a" is_pre_election: true
I20260926 21:30:27.388733 4904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fae9ce64c8845b4ae288a8435493b0d" candidate_uuid: "75421089c20f4289ba38750d161ebcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ae359b730d49f3afd42674e63ab859" is_pre_election: true
I20260926 21:30:27.388999 4639 raft_consensus.cc:2454] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75421089c20f4289ba38750d161ebcd0 in term 0.
I20260926 21:30:27.389250 4904 raft_consensus.cc:2454] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75421089c20f4289ba38750d161ebcd0 in term 0.
I20260926 21:30:27.389663 4705 leader_election.cc:302] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75421089c20f4289ba38750d161ebcd0, 92585ac4b07d41508d35cd653981fd9a; no voters:
I20260926 21:30:27.390017 4973 raft_consensus.cc:2781] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:27.390136 4973 raft_consensus.cc:482] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:27.390280 4973 raft_consensus.cc:3037] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:27.392427 4973 raft_consensus.cc:504] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
I20260926 21:30:27.392997 4973 leader_election.cc:288] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [CANDIDATE]: Term 1 election: Requested vote from peers 92585ac4b07d41508d35cd653981fd9a (127.0.97.1:34329), d9ae359b730d49f3afd42674e63ab859 (127.0.97.3:43661)
I20260926 21:30:27.393515 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fae9ce64c8845b4ae288a8435493b0d" candidate_uuid: "75421089c20f4289ba38750d161ebcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92585ac4b07d41508d35cd653981fd9a"
I20260926 21:30:27.393518 4904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fae9ce64c8845b4ae288a8435493b0d" candidate_uuid: "75421089c20f4289ba38750d161ebcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ae359b730d49f3afd42674e63ab859"
I20260926 21:30:27.393738 4904 raft_consensus.cc:3037] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:27.393728 4639 raft_consensus.cc:3037] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:27.396859 4904 raft_consensus.cc:2454] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75421089c20f4289ba38750d161ebcd0 in term 1.
I20260926 21:30:27.396859 4639 raft_consensus.cc:2454] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75421089c20f4289ba38750d161ebcd0 in term 1.
I20260926 21:30:27.397303 4705 leader_election.cc:302] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75421089c20f4289ba38750d161ebcd0, 92585ac4b07d41508d35cd653981fd9a; no voters:
I20260926 21:30:27.397541 4973 raft_consensus.cc:2781] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:27.397956 4973 raft_consensus.cc:686] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 75421089c20f4289ba38750d161ebcd0, State: Running, Role: LEADER
I20260926 21:30:27.398471 4973 consensus_queue.cc:237] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } }
W20260926 21:30:27.400274 4950 tablet.cc:2369] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:30:27.402566 4818 tablet.cc:2369] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:27.403230 4495 catalog_manager.cc:5495] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75421089c20f4289ba38750d161ebcd0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "75421089c20f4289ba38750d161ebcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75421089c20f4289ba38750d161ebcd0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43325 } health_report { overall_health: HEALTHY } } }
W20260926 21:30:27.414724 4685 tablet.cc:2369] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:27.417579 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.412512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.0.97.2:43325)
0926 21:30:27.412670 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.412674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.412690 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:30:27.412820 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.412824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.412894 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.413011 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:27.413333 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.414023 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.415767 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.416143 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.416243 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.416636 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:27.417151 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:27.417285 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.417322 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.417383 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 21:30:27.425290 4639 raft_consensus.cc:1261] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 1 FOLLOWER]: Refusing update from remote peer 75421089c20f4289ba38750d161ebcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:27.425338 4904 raft_consensus.cc:1261] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 1 FOLLOWER]: Refusing update from remote peer 75421089c20f4289ba38750d161ebcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:27.426200 4978 consensus_queue.cc:1035] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92585ac4b07d41508d35cd653981fd9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:27.426620 4973 consensus_queue.cc:1035] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9ae359b730d49f3afd42674e63ab859" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:27.438625 4981 mvcc.cc:205] Tried to move back new op lower bound from 7333716899524194304 to 7333716899428003840. Current Snapshot: MvccSnapshot[applied={T|T < 7333716899524194304}]
I20260926 21:30:27.450570 4495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:27.450811 4495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:27.454059 4495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fa35ea571cd4b2886a7726c1ebc4811: Sending DeleteTablet for 3 replicas of tablet 7fae9ce64c8845b4ae288a8435493b0d
I20260926 21:30:27.455235 4619 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fae9ce64c8845b4ae288a8435493b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44376
I20260926 21:30:27.455420 4884 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fae9ce64c8845b4ae288a8435493b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33342
I20260926 21:30:27.456101 4752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fae9ce64c8845b4ae288a8435493b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36866
I20260926 21:30:27.456249 4992 tablet_replica.cc:330] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: stopping tablet replica
I20260926 21:30:27.456672 4992 raft_consensus.cc:2229] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:27.456794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 4557
I20260926 21:30:27.457293 4992 raft_consensus.cc:2258] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:27.458084 4993 tablet_replica.cc:330] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: stopping tablet replica
I20260926 21:30:27.458458 4993 raft_consensus.cc:2229] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:27.459044 4993 raft_consensus.cc:2258] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:27.463534 4992 ts_tablet_manager.cc:1905] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:27.473383 4993 ts_tablet_manager.cc:1905] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:27.475242 4991 tablet_replica.cc:330] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: stopping tablet replica
I20260926 21:30:27.475682 4991 raft_consensus.cc:2229] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:27.476186 4991 raft_consensus.cc:2258] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:27.478374 4991 ts_tablet_manager.cc:1905] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:30:27 dist-test-slave-87bt krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:27.480825 4993 ts_tablet_manager.cc:1918] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:27.481936 4992 ts_tablet_manager.cc:1918] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:27.482185 4992 log.cc:1196] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/7fae9ce64c8845b4ae288a8435493b0d
I20260926 21:30:27.482856 4992 ts_tablet_manager.cc:1939] T 7fae9ce64c8845b4ae288a8435493b0d P d9ae359b730d49f3afd42674e63ab859: Deleting consensus metadata
I20260926 21:30:27.484577 4993 log.cc:1196] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/7fae9ce64c8845b4ae288a8435493b0d
I20260926 21:30:27.484719 4480 catalog_manager.cc:4841] TS d9ae359b730d49f3afd42674e63ab859 (127.0.97.3:43661): tablet 7fae9ce64c8845b4ae288a8435493b0d (table test-table [id=1843112e3d8a4f44b78847c7e144b18f]) successfully deleted
I20260926 21:30:27.485234 4993 ts_tablet_manager.cc:1939] T 7fae9ce64c8845b4ae288a8435493b0d P 75421089c20f4289ba38750d161ebcd0: Deleting consensus metadata
I20260926 21:30:27.486002 4991 ts_tablet_manager.cc:1918] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:27.486223 4991 log.cc:1196] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/7fae9ce64c8845b4ae288a8435493b0d
I20260926 21:30:27.486845 4991 ts_tablet_manager.cc:1939] T 7fae9ce64c8845b4ae288a8435493b0d P 92585ac4b07d41508d35cd653981fd9a: Deleting consensus metadata
I20260926 21:30:27.486958 4480 catalog_manager.cc:4841] TS 75421089c20f4289ba38750d161ebcd0 (127.0.97.2:43325): tablet 7fae9ce64c8845b4ae288a8435493b0d (table test-table [id=1843112e3d8a4f44b78847c7e144b18f]) successfully deleted
I20260926 21:30:27.488525 4480 catalog_manager.cc:4841] TS 92585ac4b07d41508d35cd653981fd9a (127.0.97.1:34329): tablet 7fae9ce64c8845b4ae288a8435493b0d (table test-table [id=1843112e3d8a4f44b78847c7e144b18f]) successfully deleted
I20260926 21:30:27.488996 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.458786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.0.97.1:34329)
0926 21:30:27.459014 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.459023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.467872 (+ 8849us) server_negotiation.cc:407] Connection header received
0926 21:30:27.468304 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.468310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.468414 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.468499 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:27.470266 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.471469 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.473309 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.473980 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.474078 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.480642 (+ 6564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:27.480664 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:27.480683 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:27.480725 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:27.485477 (+ 4752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.486350 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.486359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.486367 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.486631 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.487100 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.487105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.487108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.487549 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:27.487915 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.488213 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.488426 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:30:27.489669 4679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:27.587703 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 4557
I20260926 21:30:27.606022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 4690
Sep 26 21:30:27 dist-test-slave-87bt krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:30:27.623485 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.606874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36886 (local address 127.0.97.2:43325)
0926 21:30:27.607072 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.607078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.607099 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:30:27.607296 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.607301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.607380 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.607496 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:27.608719 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.609985 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.611369 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.612021 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.612163 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.616511 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:27.616528 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:27.616550 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:27.616597 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:27.620455 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.621099 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.621109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.621113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.621294 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.621752 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.621758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.621762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.622224 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:27.622575 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.622837 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.623062 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:30:27.624210 4812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:27.735285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 4690
I20260926 21:30:27.753501 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 4822
Sep 26 21:30:27 dist-test-slave-87bt krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:30:27.768040 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:27.754232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.0.97.3:43661)
0926 21:30:27.754442 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:27.754450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:27.754538 (+ 88us) server_negotiation.cc:407] Connection header received
0926 21:30:27.754751 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:27.754757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:27.754832 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:27.754949 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:27.756177 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.756921 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.758141 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:27.758609 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:27.758743 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:27.762788 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:27.762801 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:27.762817 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:27.762848 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:27.765327 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.765997 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.766004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.766006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.766133 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:27.766576 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:27.766588 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:27.766590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:27.766922 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:27.767207 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:27.767508 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 21:30:27.767733 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:30:27.768702 4944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:27.856772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 4822
I20260926 21:30:27.872830 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 4465
I20260926 21:30:27.873935 4526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:27.977571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 4465
2026-09-26T21:30:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4118 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T21:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:28Z Disabled control of system clock
I20260926 21:30:28.043998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45617
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 21:30:28.365072 5002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:28.365367 5002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:28.365450 5002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:28.372736 5002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:28.372846 5002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:28.372884 5002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:28.372939 5002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:28.372982 5002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:28.382258 5002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:28.383611 5002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:28.385362 5002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:28.397012 5007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:28.397038 5008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:28.397607 5010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:28.398032 5002 server_base.cc:1034] running on GCE node
I20260926 21:30:28.398864 5002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:28.400231 5002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:28.401486 5002 hybrid_clock.cc:648] HybridClock initialized: now 1790458228401444 us; error 52 us; skew 500 ppm
I20260926 21:30:28.401877 5002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:28.404186 5002 webserver.cc:460] Webserver started at http://127.0.97.62:42621/ using document root <none> and password file <none>
I20260926 21:30:28.404894 5002 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:28.405027 5002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:28.405352 5002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:28.408283 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "c5ba2503b4194fe78f52d50f78f49334"
format_stamp: "Formatted at 2026-09-26 21:30:28 on dist-test-slave-87bt"
server_key: "4147b80f807dcee4bbfd2d67530f0b29"
server_key_iv: "282133eddb525052ace95d440379866c"
server_key_version: "encryptionkey@0"
I20260926 21:30:28.409094 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5ba2503b4194fe78f52d50f78f49334"
format_stamp: "Formatted at 2026-09-26 21:30:28 on dist-test-slave-87bt"
server_key: "4147b80f807dcee4bbfd2d67530f0b29"
server_key_iv: "282133eddb525052ace95d440379866c"
server_key_version: "encryptionkey@0"
I20260926 21:30:28.414165 5002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 21:30:28.417569 5016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:28.418915 5002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:28.419092 5002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "c5ba2503b4194fe78f52d50f78f49334"
format_stamp: "Formatted at 2026-09-26 21:30:28 on dist-test-slave-87bt"
server_key: "4147b80f807dcee4bbfd2d67530f0b29"
server_key_iv: "282133eddb525052ace95d440379866c"
server_key_version: "encryptionkey@0"
I20260926 21:30:28.419312 5002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:28.424901 5002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:28.425655 5002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:28.425941 5002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:28.444481 5002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45617
I20260926 21:30:28.444502 5067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45617 every 8 connection(s)
I20260926 21:30:28.445814 5002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:28.449386 5068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:28.452498 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5002
I20260926 21:30:28.452689 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:28.453284 388 external_mini_cluster.cc:1428] Setting key 6b6d9225aa57e4ce91d7074d79252103
I20260926 21:30:28.462116 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:28.455288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.0.97.62:45617)
0926 21:30:28.456326 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:28.456340 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:28.456392 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:30:28.458087 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:28.458121 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:28.458723 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:28.459168 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:28.460178 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:28.460186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:28.460202 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:28.460218 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:28.460293 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:28.460315 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:28.460543 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:28.460667 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 21:30:28.460931 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":321,"threads_started":1}
I20260926 21:30:28.462733 5068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334: Bootstrap starting.
I20260926 21:30:28.465821 5068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:28.466944 5068 log.cc:824] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:28.469376 5068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334: No bootstrap required, opened a new log
I20260926 21:30:28.477031 5068 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ba2503b4194fe78f52d50f78f49334" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45617 } }
I20260926 21:30:28.477336 5068 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:28.477411 5068 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ba2503b4194fe78f52d50f78f49334, State: Initialized, Role: FOLLOWER
I20260926 21:30:28.477861 5068 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ba2503b4194fe78f52d50f78f49334" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45617 } }
I20260926 21:30:28.478049 5068 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:28.478142 5068 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:28.478261 5068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:28.480273 5068 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ba2503b4194fe78f52d50f78f49334" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45617 } }
I20260926 21:30:28.480682 5068 leader_election.cc:302] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5ba2503b4194fe78f52d50f78f49334; no voters:
I20260926 21:30:28.481182 5068 leader_election.cc:288] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:28.481328 5073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:28.482033 5073 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [term 1 LEADER]: Becoming Leader. State: Replica: c5ba2503b4194fe78f52d50f78f49334, State: Running, Role: LEADER
I20260926 21:30:28.482642 5073 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ba2503b4194fe78f52d50f78f49334" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45617 } }
I20260926 21:30:28.483219 5068 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:28.485466 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5ba2503b4194fe78f52d50f78f49334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ba2503b4194fe78f52d50f78f49334" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45617 } } }
I20260926 21:30:28.485898 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:28.486647 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ba2503b4194fe78f52d50f78f49334. Latest consensus state: current_term: 1 leader_uuid: "c5ba2503b4194fe78f52d50f78f49334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ba2503b4194fe78f52d50f78f49334" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45617 } } }
I20260926 21:30:28.487105 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:28.487694 5079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:28.492950 5079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:28.501667 5079 catalog_manager.cc:1347] Generated new cluster ID: 88912874b2e24aaab5ef100feb415510
I20260926 21:30:28.501786 5079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:28.528535 5079 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:28.528975 5079 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:28.536738 5079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334: Generated new TSK 0
I20260926 21:30:28.537581 5079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:30:28.544051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45617
--builtin_ntp_servers=127.0.97.20:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 21:30:28.893700 5092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:28.894035 5092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:28.894207 5092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:28.901433 5092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:28.901571 5092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:28.901755 5092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:28.910907 5092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:28.912202 5092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:28.914003 5092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:28.927058 5097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:28.927130 5100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:28.927212 5098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:28.928181 5092 server_base.cc:1034] running on GCE node
I20260926 21:30:28.928753 5092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:28.929759 5092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:28.930956 5092 hybrid_clock.cc:648] HybridClock initialized: now 1790458228930912 us; error 29 us; skew 500 ppm
I20260926 21:30:28.931284 5092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:28.933527 5092 webserver.cc:460] Webserver started at http://127.0.97.1:40201/ using document root <none> and password file <none>
I20260926 21:30:28.934214 5092 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:28.934331 5092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:28.934698 5092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:28.937522 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "419570a3ab8941b681d575ae47962e99"
format_stamp: "Formatted at 2026-09-26 21:30:28 on dist-test-slave-87bt"
server_key: "91c3cebd5ebab9922f59423768745648"
server_key_iv: "df9985b0e29ba624da4dcc7cf8513dee"
server_key_version: "encryptionkey@0"
I20260926 21:30:28.938316 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "419570a3ab8941b681d575ae47962e99"
format_stamp: "Formatted at 2026-09-26 21:30:28 on dist-test-slave-87bt"
server_key: "91c3cebd5ebab9922f59423768745648"
server_key_iv: "df9985b0e29ba624da4dcc7cf8513dee"
server_key_version: "encryptionkey@0"
I20260926 21:30:28.943244 5092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 21:30:28.946662 5106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:28.948172 5092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:30:28.948347 5092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "419570a3ab8941b681d575ae47962e99"
format_stamp: "Formatted at 2026-09-26 21:30:28 on dist-test-slave-87bt"
server_key: "91c3cebd5ebab9922f59423768745648"
server_key_iv: "df9985b0e29ba624da4dcc7cf8513dee"
server_key_version: "encryptionkey@0"
I20260926 21:30:28.948645 5092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:28.954519 5092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:28.955268 5092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:28.955484 5092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:28.956305 5092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:28.957605 5092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:28.957697 5092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:28.957777 5092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:28.957831 5092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:28.990947 5092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45987
I20260926 21:30:28.990999 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45987 every 8 connection(s)
I20260926 21:30:28.992416 5092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:28.998018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5092
I20260926 21:30:28.998193 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:28.998899 388 external_mini_cluster.cc:1428] Setting key bbe9e497749093b80573681d425e7c62
I20260926 21:30:29.000635 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:28.994986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55925 (local address 127.0.97.62:45617)
0926 21:30:28.995423 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:28.995427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:28.996561 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 21:30:28.998243 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:28.998248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:28.998315 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:28.998431 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:28.999884 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:28.999890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:28.999895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:28.999913 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:28.999959 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:28.999971 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:29.000095 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:29.000297 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 21:30:29.000362 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":240,"threads_started":1}
I20260926 21:30:29.001394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45617
--builtin_ntp_servers=127.0.97.20:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 21:30:29.001534 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:28.995540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45617 (local address 127.0.97.1:55925)
0926 21:30:28.996407 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:28.996441 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:28.997958 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:28.998686 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:28.998694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:28.998849 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:28.999592 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:30:28.999594 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:28.999630 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:30:28.999713 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:29.000065 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:29.000084 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:29.000093 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 21:30:29.000364 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 21:30:29.000601 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":261,"threads_started":1}
I20260926 21:30:29.003031 5219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45617
I20260926 21:30:29.003324 5219 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:29.003871 5219 heartbeater.cc:511] Master 127.0.97.62:45617 requested a full tablet report, sending...
I20260926 21:30:29.006075 5033 ts_manager.cc:194] Registered new tserver with Master: 419570a3ab8941b681d575ae47962e99 (127.0.97.1:45987)
W20260926 21:30:29.323577 5223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:29.323907 5223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:29.324069 5223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:29.331470 5223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:29.331624 5223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:29.331795 5223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:29.341037 5223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:29.342350 5223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:29.344220 5223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:29.358263 5229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:29.358287 5228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:29.358318 5231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:29.358982 5223 server_base.cc:1034] running on GCE node
I20260926 21:30:29.359735 5223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:29.360741 5223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:29.361934 5223 hybrid_clock.cc:648] HybridClock initialized: now 1790458229361871 us; error 67 us; skew 500 ppm
I20260926 21:30:29.362258 5223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:29.364590 5223 webserver.cc:460] Webserver started at http://127.0.97.2:43143/ using document root <none> and password file <none>
I20260926 21:30:29.365294 5223 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:29.365410 5223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:29.365746 5223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:29.368772 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "8c7d93a378a64aa29e20231782016162"
format_stamp: "Formatted at 2026-09-26 21:30:29 on dist-test-slave-87bt"
server_key: "9a4fa0e988e3304c779368cc2ff1ddd1"
server_key_iv: "ecfd78200aea3cff6e81b12f1733c8d3"
server_key_version: "encryptionkey@0"
I20260926 21:30:29.369612 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8c7d93a378a64aa29e20231782016162"
format_stamp: "Formatted at 2026-09-26 21:30:29 on dist-test-slave-87bt"
server_key: "9a4fa0e988e3304c779368cc2ff1ddd1"
server_key_iv: "ecfd78200aea3cff6e81b12f1733c8d3"
server_key_version: "encryptionkey@0"
I20260926 21:30:29.374744 5223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:30:29.378295 5237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:29.379755 5223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:29.379931 5223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "8c7d93a378a64aa29e20231782016162"
format_stamp: "Formatted at 2026-09-26 21:30:29 on dist-test-slave-87bt"
server_key: "9a4fa0e988e3304c779368cc2ff1ddd1"
server_key_iv: "ecfd78200aea3cff6e81b12f1733c8d3"
server_key_version: "encryptionkey@0"
I20260926 21:30:29.380175 5223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:29.386188 5223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:29.387092 5223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:29.387365 5223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:29.388295 5223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:29.389791 5223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:29.389922 5223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:29.390046 5223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:29.390132 5223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:29.422309 5223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43553
I20260926 21:30:29.422333 5349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43553 every 8 connection(s)
I20260926 21:30:29.423764 5223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:29.431399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5223
I20260926 21:30:29.431566 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:29.431772 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.426303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52699 (local address 127.0.97.62:45617)
0926 21:30:29.426537 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:29.426544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:29.427934 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 21:30:29.429528 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:29.429532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:29.429582 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:29.429673 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:29.431022 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:29.431030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:29.431037 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:29.431059 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:29.431095 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:29.431112 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:29.431230 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:29.431401 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 21:30:29.431463 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:30:29.432096 388 external_mini_cluster.cc:1428] Setting key b0658ac3a2c91a665db942e605dbf7fb
I20260926 21:30:29.432718 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.426899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45617 (local address 127.0.97.2:52699)
0926 21:30:29.427780 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:29.427821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:29.429291 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:29.429866 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:29.429875 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:29.429963 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:29.430691 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:30:29.430693 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:29.430728 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:30:29.430810 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:29.431216 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:29.431222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:29.431228 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:30:29.431425 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 21:30:29.431725 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":281,"threads_started":1}
I20260926 21:30:29.434074 5350 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45617
I20260926 21:30:29.434372 5350 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:29.434792 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45617
--builtin_ntp_servers=127.0.97.20:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 21:30:29.435029 5350 heartbeater.cc:511] Master 127.0.97.62:45617 requested a full tablet report, sending...
I20260926 21:30:29.436379 5033 ts_manager.cc:194] Registered new tserver with Master: 8c7d93a378a64aa29e20231782016162 (127.0.97.2:43553)
W20260926 21:30:29.762231 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:29.762566 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:29.762662 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:29.769742 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:29.769830 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:29.769937 5353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:29.778810 5353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:29.780001 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:29.781620 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:29.796154 5359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:29.796162 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:29.796952 5358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:29.796933 5353 server_base.cc:1034] running on GCE node
I20260926 21:30:29.797703 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:29.798686 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:29.799901 5353 hybrid_clock.cc:648] HybridClock initialized: now 1790458229799858 us; error 62 us; skew 500 ppm
I20260926 21:30:29.800216 5353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:29.802466 5353 webserver.cc:460] Webserver started at http://127.0.97.3:46601/ using document root <none> and password file <none>
I20260926 21:30:29.803177 5353 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:29.803303 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:29.803608 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:29.806366 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "c6d2f5aa35df446d926e254adedeca28"
format_stamp: "Formatted at 2026-09-26 21:30:29 on dist-test-slave-87bt"
server_key: "f218543184e40281f61a7458762e9c3b"
server_key_iv: "3bd26142ca5f2b751667fc68974adfd7"
server_key_version: "encryptionkey@0"
I20260926 21:30:29.807201 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c6d2f5aa35df446d926e254adedeca28"
format_stamp: "Formatted at 2026-09-26 21:30:29 on dist-test-slave-87bt"
server_key: "f218543184e40281f61a7458762e9c3b"
server_key_iv: "3bd26142ca5f2b751667fc68974adfd7"
server_key_version: "encryptionkey@0"
I20260926 21:30:29.812052 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:30:29.815328 5367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:29.816592 5353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:29.816766 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "c6d2f5aa35df446d926e254adedeca28"
format_stamp: "Formatted at 2026-09-26 21:30:29 on dist-test-slave-87bt"
server_key: "f218543184e40281f61a7458762e9c3b"
server_key_iv: "3bd26142ca5f2b751667fc68974adfd7"
server_key_version: "encryptionkey@0"
I20260926 21:30:29.816987 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:29.822690 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:29.823452 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:29.823716 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:29.824599 5353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:29.825934 5353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:29.826066 5353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:29.826201 5353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:29.826308 5353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:29.857983 5353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43961
I20260926 21:30:29.858034 5479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43961 every 8 connection(s)
I20260926 21:30:29.859337 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:29.863933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5353
I20260926 21:30:29.864099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:29.864646 388 external_mini_cluster.cc:1428] Setting key d8327e1baece28abdc305e725c04b611
I20260926 21:30:29.868132 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.861852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38877 (local address 127.0.97.62:45617)
0926 21:30:29.862060 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:29.862067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:29.863472 (+ 1405us) server_negotiation.cc:407] Connection header received
0926 21:30:29.865767 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:29.865773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:29.865837 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:29.865957 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:29.867475 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:29.867483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:29.867487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:29.867504 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:29.867543 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:29.867558 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:29.867657 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:29.867872 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 21:30:29.867915 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:30:29.868999 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.862391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45617 (local address 127.0.97.3:38877)
0926 21:30:29.863316 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:29.863352 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:29.865482 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:29.866256 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:29.866268 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:29.866393 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:29.867153 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:30:29.867155 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:29.867192 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:30:29.867312 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:29.867670 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:29.867682 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:29.867690 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 21:30:29.867923 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 21:30:29.868135 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":267,"threads_started":1}
I20260926 21:30:29.870134 5480 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45617
I20260926 21:30:29.870410 5480 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:29.870975 5480 heartbeater.cc:511] Master 127.0.97.62:45617 requested a full tablet report, sending...
I20260926 21:30:29.872215 5033 ts_manager.cc:194] Registered new tserver with Master: c6d2f5aa35df446d926e254adedeca28 (127.0.97.3:43961)
I20260926 21:30:29.879621 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:30:29.886241 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.883839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.0.97.62:45617)
0926 21:30:29.884040 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:29.884059 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:29.884482 (+ 423us) server_negotiation.cc:407] Connection header received
0926 21:30:29.884660 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:29.884663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:29.884709 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:29.884807 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:29.885682 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:29.885686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:29.885690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:29.885703 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:29.885728 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:29.885738 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:29.885827 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:29.885912 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 21:30:29.885954 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:30:29.889796 5033 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:29.896183 5033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:29.914880 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.907383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43961 (local address 127.0.0.1:53018)
0926 21:30:29.908873 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:29.908908 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:29.909033 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:29.912570 (+ 3537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:29.912574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:29.912599 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:29.913540 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:30:29.913544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:29.913559 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:30:29.913600 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:29.914089 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:29.914099 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:29.914106 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 21:30:29.914281 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 21:30:29.914390 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"thread_start_us":1090,"threads_started":1}
I20260926 21:30:29.914986 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.909500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.0.97.2:43553)
0926 21:30:29.909970 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:29.909976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:29.911865 (+ 1889us) server_negotiation.cc:407] Connection header received
0926 21:30:29.912099 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:29.912107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:29.912304 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:29.912431 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:29.914037 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:29.914046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:29.914060 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:29.914080 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:29.914134 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:29.914158 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:29.914282 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:29.914552 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 21:30:29.914658 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":228,"threads_started":1}
I20260926 21:30:29.915340 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.908374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.0.97.1:45987)
0926 21:30:29.911535 (+ 3161us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:29.911559 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:29.911593 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:30:29.911666 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:29.911673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:29.911925 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:29.913511 (+ 1586us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:29.914303 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:29.914312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:29.914327 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:29.914343 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:29.914391 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:29.914409 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:29.914566 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:29.914970 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 21:30:29.915070 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2990,"thread_start_us":2888,"threads_started":1}
I20260926 21:30:29.915625 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.909461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43553 (local address 127.0.0.1:57748)
0926 21:30:29.911761 (+ 2300us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:29.911777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:29.911937 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:29.913079 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:29.913086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:29.913114 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:29.913815 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:30:29.913818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:29.913831 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:30:29.913874 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:29.914369 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:29.914373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:29.914377 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 21:30:29.915200 (+ 823us) client_negotiation.cc:240] Negotiation successful
0926 21:30:29.915245 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"thread_start_us":1205,"threads_started":1}
I20260926 21:30:29.915830 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.907831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.0.97.3:43961)
0926 21:30:29.912174 (+ 4343us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:29.912180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:29.912206 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:30:29.912257 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:29.912263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:29.912415 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:29.912573 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:29.913760 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:29.913772 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:29.913789 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:29.913808 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:29.913860 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:29.913880 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:29.914040 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:29.915353 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0926 21:30:29.915453 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4210,"thread_start_us":4039,"threads_started":1}
I20260926 21:30:29.916007 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:29.907547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.0.1:53716)
0926 21:30:29.910979 (+ 3432us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:29.910996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:29.911082 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:29.912119 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:29.912126 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:29.912213 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:29.913245 (+ 1032us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:30:29.913248 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:29.913278 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:30:29.913379 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:29.914616 (+ 1237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:29.914620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:29.914624 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 21:30:29.914775 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 21:30:29.914827 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3265,"thread_start_us":3997,"threads_started":1}
I20260926 21:30:29.919380 5285 tablet_service.cc:1467] Processing CreateTablet for tablet c24186cd348740dd9afe375f993b35c6 (DEFAULT_TABLE table=test-table [id=3adb8562977349cfab047dc546dc9158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:29.919410 5154 tablet_service.cc:1467] Processing CreateTablet for tablet c24186cd348740dd9afe375f993b35c6 (DEFAULT_TABLE table=test-table [id=3adb8562977349cfab047dc546dc9158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:29.919404 5415 tablet_service.cc:1467] Processing CreateTablet for tablet c24186cd348740dd9afe375f993b35c6 (DEFAULT_TABLE table=test-table [id=3adb8562977349cfab047dc546dc9158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:29.920629 5285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24186cd348740dd9afe375f993b35c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:29.920696 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24186cd348740dd9afe375f993b35c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:29.920696 5154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24186cd348740dd9afe375f993b35c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:29.932519 5498 tablet_bootstrap.cc:492] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Bootstrap starting.
I20260926 21:30:29.934865 5499 tablet_bootstrap.cc:492] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Bootstrap starting.
I20260926 21:30:29.935819 5500 tablet_bootstrap.cc:492] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Bootstrap starting.
I20260926 21:30:29.937350 5498 tablet_bootstrap.cc:654] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:29.938416 5499 tablet_bootstrap.cc:654] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:29.938753 5498 log.cc:824] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:29.938962 5500 tablet_bootstrap.cc:654] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:29.939658 5499 log.cc:824] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:29.940187 5500 log.cc:824] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:29.941784 5498 tablet_bootstrap.cc:492] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: No bootstrap required, opened a new log
I20260926 21:30:29.942039 5498 ts_tablet_manager.cc:1397] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 21:30:29.942664 5499 tablet_bootstrap.cc:492] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: No bootstrap required, opened a new log
I20260926 21:30:29.942927 5499 ts_tablet_manager.cc:1397] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260926 21:30:29.943075 5500 tablet_bootstrap.cc:492] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: No bootstrap required, opened a new log
I20260926 21:30:29.943306 5500 ts_tablet_manager.cc:1397] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:30:29.949323 5498 raft_consensus.cc:348] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:29.949707 5498 raft_consensus.cc:374] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:29.949828 5498 raft_consensus.cc:729] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7d93a378a64aa29e20231782016162, State: Initialized, Role: FOLLOWER
I20260926 21:30:29.950500 5498 consensus_queue.cc:260] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:29.951555 5350 heartbeater.cc:503] Master 127.0.97.62:45617 was elected leader, sending a full tablet report...
I20260926 21:30:29.952263 5498 ts_tablet_manager.cc:1428] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260926 21:30:29.953552 5499 raft_consensus.cc:348] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:29.953619 5500 raft_consensus.cc:348] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:29.954012 5500 raft_consensus.cc:374] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:29.954041 5499 raft_consensus.cc:374] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:29.954138 5500 raft_consensus.cc:729] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419570a3ab8941b681d575ae47962e99, State: Initialized, Role: FOLLOWER
I20260926 21:30:29.954154 5499 raft_consensus.cc:729] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d2f5aa35df446d926e254adedeca28, State: Initialized, Role: FOLLOWER
I20260926 21:30:29.954766 5500 consensus_queue.cc:260] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:29.954766 5499 consensus_queue.cc:260] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:29.956296 5219 heartbeater.cc:503] Master 127.0.97.62:45617 was elected leader, sending a full tablet report...
I20260926 21:30:29.956600 5499 ts_tablet_manager.cc:1428] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 21:30:29.956827 5500 ts_tablet_manager.cc:1428] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 21:30:29.957370 5480 heartbeater.cc:503] Master 127.0.97.62:45617 was elected leader, sending a full tablet report...
W20260926 21:30:29.994727 5220 tablet.cc:2369] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:30:30.111024 5481 tablet.cc:2369] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:30:30.175863 5351 tablet.cc:2369] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:30.207528 5504 raft_consensus.cc:482] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:30.207818 5504 raft_consensus.cc:504] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:30.209530 5504 leader_election.cc:288] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d2f5aa35df446d926e254adedeca28 (127.0.97.3:43961), 419570a3ab8941b681d575ae47962e99 (127.0.97.1:45987)
I20260926 21:30:30.212833 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.210112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60273 (local address 127.0.97.1:45987)
0926 21:30:30.210299 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:30.210307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:30.210784 (+ 477us) server_negotiation.cc:407] Connection header received
0926 21:30:30.210988 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:30.210998 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:30.211048 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:30.211150 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:30.212186 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:30.212192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:30.212196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:30.212218 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:30.212253 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:30.212264 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:30.212381 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:30.212471 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 21:30:30.212543 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:30:30.212877 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.209869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57325 (local address 127.0.97.3:43961)
0926 21:30:30.210063 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:30.210069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:30.211047 (+ 978us) server_negotiation.cc:407] Connection header received
0926 21:30:30.211226 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:30.211236 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:30.211282 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:30.211388 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:30.212272 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:30.212277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:30.212282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:30.212301 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:30.212333 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:30.212344 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:30.212471 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:30.212560 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 21:30:30.212620 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:30:30.213047 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.209873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.97.2:60273)
0926 21:30:30.210636 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.210652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:30.210843 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:30.211172 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:30.211177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:30.211211 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:30.211936 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:30:30.211942 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:30.211961 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:30:30.212021 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:30.212365 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:30.212370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:30.212375 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:30:30.212519 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 21:30:30.212571 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":389,"threads_started":1}
I20260926 21:30:30.213099 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.209871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43961 (local address 127.0.97.2:57325)
0926 21:30:30.210938 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.210952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:30.211086 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:30.211411 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:30.211444 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:30.211467 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:30.212094 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:30:30.212096 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:30.212107 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:30:30.212138 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:30.212436 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:30.212441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:30.212446 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:30:30.212588 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 21:30:30.212636 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":210,"thread_start_us":512,"threads_started":1}
I20260926 21:30:30.214098 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24186cd348740dd9afe375f993b35c6" candidate_uuid: "8c7d93a378a64aa29e20231782016162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419570a3ab8941b681d575ae47962e99" is_pre_election: true
I20260926 21:30:30.214125 5435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24186cd348740dd9afe375f993b35c6" candidate_uuid: "8c7d93a378a64aa29e20231782016162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d2f5aa35df446d926e254adedeca28" is_pre_election: true
I20260926 21:30:30.214638 5435 raft_consensus.cc:2454] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c7d93a378a64aa29e20231782016162 in term 0.
I20260926 21:30:30.214638 5174 raft_consensus.cc:2454] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c7d93a378a64aa29e20231782016162 in term 0.
I20260926 21:30:30.215327 5240 leader_election.cc:302] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 419570a3ab8941b681d575ae47962e99, 8c7d93a378a64aa29e20231782016162, c6d2f5aa35df446d926e254adedeca28; no voters:
I20260926 21:30:30.215736 5504 raft_consensus.cc:2781] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:30.215879 5504 raft_consensus.cc:482] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:30.215983 5504 raft_consensus.cc:3037] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:30.218127 5504 raft_consensus.cc:504] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:30.218815 5504 leader_election.cc:288] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [CANDIDATE]: Term 1 election: Requested vote from peers c6d2f5aa35df446d926e254adedeca28 (127.0.97.3:43961), 419570a3ab8941b681d575ae47962e99 (127.0.97.1:45987)
I20260926 21:30:30.219304 5435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24186cd348740dd9afe375f993b35c6" candidate_uuid: "8c7d93a378a64aa29e20231782016162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d2f5aa35df446d926e254adedeca28"
I20260926 21:30:30.219337 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24186cd348740dd9afe375f993b35c6" candidate_uuid: "8c7d93a378a64aa29e20231782016162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419570a3ab8941b681d575ae47962e99"
I20260926 21:30:30.219511 5435 raft_consensus.cc:3037] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:30.219560 5174 raft_consensus.cc:3037] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:30.221882 5174 raft_consensus.cc:2454] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c7d93a378a64aa29e20231782016162 in term 1.
I20260926 21:30:30.221886 5435 raft_consensus.cc:2454] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c7d93a378a64aa29e20231782016162 in term 1.
I20260926 21:30:30.222332 5240 leader_election.cc:302] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 419570a3ab8941b681d575ae47962e99, 8c7d93a378a64aa29e20231782016162, c6d2f5aa35df446d926e254adedeca28; no voters:
I20260926 21:30:30.222661 5504 raft_consensus.cc:2781] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:30.223114 5504 raft_consensus.cc:686] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 1 LEADER]: Becoming Leader. State: Replica: 8c7d93a378a64aa29e20231782016162, State: Running, Role: LEADER
I20260926 21:30:30.223750 5504 consensus_queue.cc:237] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } }
I20260926 21:30:30.228554 5033 catalog_manager.cc:5495] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c7d93a378a64aa29e20231782016162 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8c7d93a378a64aa29e20231782016162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c7d93a378a64aa29e20231782016162" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43553 } health_report { overall_health: HEALTHY } } }
I20260926 21:30:30.279986 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.278175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.97.2:43553)
0926 21:30:30.278483 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:30.278488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:30.278512 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:30:30.278597 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:30.278601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:30.278673 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:30.278768 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:30.279534 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:30.279539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:30.279542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:30.279556 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:30.279584 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:30.279596 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:30.279691 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:30.279753 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 21:30:30.279796 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260926 21:30:30.288367 5435 raft_consensus.cc:1261] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 1 FOLLOWER]: Refusing update from remote peer 8c7d93a378a64aa29e20231782016162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:30.288367 5174 raft_consensus.cc:1261] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 1 FOLLOWER]: Refusing update from remote peer 8c7d93a378a64aa29e20231782016162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:30.289435 5509 consensus_queue.cc:1035] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2f5aa35df446d926e254adedeca28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:30.289825 5504 consensus_queue.cc:1035] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419570a3ab8941b681d575ae47962e99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:30.301776 5513 mvcc.cc:205] Tried to move back new op lower bound from 7333716911250452480 to 7333716911000530944. Current Snapshot: MvccSnapshot[applied={T|T < 7333716911250452480}]
I20260926 21:30:30.317567 5033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:30.317848 5033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:30.321688 5033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5ba2503b4194fe78f52d50f78f49334: Sending DeleteTablet for 3 replicas of tablet c24186cd348740dd9afe375f993b35c6
I20260926 21:30:30.322805 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet c24186cd348740dd9afe375f993b35c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:30 UTC) from {username='slave'} at 127.0.0.1:53018
I20260926 21:30:30.323004 5285 tablet_service.cc:1514] Processing DeleteTablet for tablet c24186cd348740dd9afe375f993b35c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:30 UTC) from {username='slave'} at 127.0.0.1:57748
I20260926 21:30:30.323679 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet c24186cd348740dd9afe375f993b35c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:30 UTC) from {username='slave'} at 127.0.0.1:53716
I20260926 21:30:30.324848 5522 tablet_replica.cc:330] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: stopping tablet replica
I20260926 21:30:30.325006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5092
I20260926 21:30:30.325295 5522 raft_consensus.cc:2229] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:30.325773 5522 raft_consensus.cc:2258] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:30.328305 5522 ts_tablet_manager.cc:1905] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:30.337504 5522 ts_tablet_manager.cc:1918] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:30.337697 5522 log.cc:1196] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/c24186cd348740dd9afe375f993b35c6
I20260926 21:30:30.338277 5522 ts_tablet_manager.cc:1939] T c24186cd348740dd9afe375f993b35c6 P c6d2f5aa35df446d926e254adedeca28: Deleting consensus metadata
I20260926 21:30:30.339076 5524 tablet_replica.cc:330] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: stopping tablet replica
I20260926 21:30:30.339615 5524 raft_consensus.cc:2229] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:30.340227 5524 raft_consensus.cc:2258] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:30.342051 5524 ts_tablet_manager.cc:1905] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:30.343209 5523 tablet_replica.cc:330] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: stopping tablet replica
I20260926 21:30:30.343627 5523 raft_consensus.cc:2229] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:30.344290 5523 raft_consensus.cc:2258] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:30.345881 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.326890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.0.97.1:45987)
0926 21:30:30.327087 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:30.327096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:30.341235 (+ 14139us) server_negotiation.cc:407] Connection header received
0926 21:30:30.341965 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:30.341971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:30.342062 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:30.342178 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:30.345066 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:30.345072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:30.345076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:30.345097 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:30.345133 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:30.345149 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:30.345276 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:30.345513 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 21:30:30.345577 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:30:30.346753 5020 catalog_manager.cc:4841] TS c6d2f5aa35df446d926e254adedeca28 (127.0.97.3:43961): tablet c24186cd348740dd9afe375f993b35c6 (table test-table [id=3adb8562977349cfab047dc546dc9158]) successfully deleted
I20260926 21:30:30.347813 5214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:30.347846 5523 ts_tablet_manager.cc:1905] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:30.355702 5523 ts_tablet_manager.cc:1918] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:30.355899 5523 log.cc:1196] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/c24186cd348740dd9afe375f993b35c6
I20260926 21:30:30.356420 5523 ts_tablet_manager.cc:1939] T c24186cd348740dd9afe375f993b35c6 P 8c7d93a378a64aa29e20231782016162: Deleting consensus metadata
I20260926 21:30:30.357920 5017 catalog_manager.cc:4841] TS 8c7d93a378a64aa29e20231782016162 (127.0.97.2:43553): tablet c24186cd348740dd9afe375f993b35c6 (table test-table [id=3adb8562977349cfab047dc546dc9158]) successfully deleted
I20260926 21:30:30.498020 5524 ts_tablet_manager.cc:1918] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:30.498287 5524 log.cc:1196] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/c24186cd348740dd9afe375f993b35c6
I20260926 21:30:30.498759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5092
I20260926 21:30:30.498896 5524 ts_tablet_manager.cc:1939] T c24186cd348740dd9afe375f993b35c6 P 419570a3ab8941b681d575ae47962e99: Deleting consensus metadata
I20260926 21:30:30.519691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5223
W20260926 21:30:30.519692 5019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45987 (error 108)
I20260926 21:30:30.521708 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.520978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.0.1:53736)
0926 21:30:30.521397 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.521524 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"threads_started":1}
W20260926 21:30:30.522385 5019 catalog_manager.cc:4584] TS 419570a3ab8941b681d575ae47962e99 (127.0.97.1:45987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c24186cd348740dd9afe375f993b35c6: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45987: connect: Connection refused (error 111)
I20260926 21:30:30.523191 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.520909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.0.97.2:43553)
0926 21:30:30.521232 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:30.521239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:30.521258 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:30:30.521362 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:30.521367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:30.521428 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:30.521535 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:30.522533 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:30.522539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:30.522544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:30.522567 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:30.522616 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:30.522633 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:30.522760 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:30.522876 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 21:30:30.522932 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260926 21:30:30.523967 5345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:30.576124 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.575615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.0.1:53746)
0926 21:30:30.575844 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.575962 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 21:30:30.577176 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.576782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.0.1:53762)
0926 21:30:30.576934 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.577019 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 21:30:30.633299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5223
I20260926 21:30:30.637861 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.637316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.0.1:53774)
0926 21:30:30.637565 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.637678 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 21:30:30.639053 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.638590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.0.1:53778)
0926 21:30:30.638773 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.638860 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 21:30:30.651136 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5353
I20260926 21:30:30.654139 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.651882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.0.97.3:43961)
0926 21:30:30.652081 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:30.652087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:30.652190 (+ 103us) server_negotiation.cc:407] Connection header received
0926 21:30:30.652407 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:30.652412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:30.652490 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:30.652599 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:30.653506 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:30.653512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:30.653516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:30.653540 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:30.653568 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:30.653580 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:30.653716 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:30.653793 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 21:30:30.653854 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:30:30.654783 5475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:30.720170 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.719614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.0.1:53786)
0926 21:30:30.719872 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.719988 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 21:30:30.721335 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:30.720849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45987 (local address 127.0.0.1:53790)
0926 21:30:30.721058 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:30.721160 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 21:30:30.742337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5353
I20260926 21:30:30.758208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5002
I20260926 21:30:30.759426 5063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:30.868921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5002
2026-09-26T21:30:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2889 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 21:30:30.909480 388 mini_oidc.cc:152] Starting JWKS server
I20260926 21:30:30.912009 388 webserver.cc:460] Webserver started at https://127.0.0.1:45383/ using document root <none> and password file <none>
I20260926 21:30:30.912322 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 21:30:30.912935 388 webserver.cc:460] Webserver started at http://127.0.0.1:43573/ using document root <none> and password file <none>
2026-09-26T21:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:30Z Disabled control of system clock
I20260926 21:30:30.939206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36945
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36945
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45383/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/testchainca.cert with env {}
W20260926 21:30:31.252749 5537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:31.253064 5537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:31.253211 5537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:31.260368 5537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:31.260488 5537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:31.260571 5537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:31.260670 5537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:31.260731 5537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 21:30:31.260809 5537 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 21:30:31.260870 5537 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45383/jwks.json
I20260926 21:30:31.270771 5537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45383/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:31.272059 5537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:31.273351 5537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:31.284874 5542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:31.284785 5543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:31.284849 5545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:31.285945 5537 server_base.cc:1034] running on GCE node
I20260926 21:30:31.286747 5537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:31.288028 5537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:31.289219 5537 hybrid_clock.cc:648] HybridClock initialized: now 1790458231289182 us; error 29 us; skew 500 ppm
I20260926 21:30:31.289572 5537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:31.291706 5537 webserver.cc:460] Webserver started at http://127.0.97.62:45677/ using document root <none> and password file <none>
I20260926 21:30:31.292371 5537 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:31.292503 5537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:31.292812 5537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:31.295645 5537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "7041e794183d49379ac3886fc4345c28"
format_stamp: "Formatted at 2026-09-26 21:30:31 on dist-test-slave-87bt"
server_key: "d053dce0b931384886c969687eff4ef6"
server_key_iv: "88a278b0ed6e08048c27151efa037ee0"
server_key_version: "encryptionkey@0"
I20260926 21:30:31.296427 5537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "7041e794183d49379ac3886fc4345c28"
format_stamp: "Formatted at 2026-09-26 21:30:31 on dist-test-slave-87bt"
server_key: "d053dce0b931384886c969687eff4ef6"
server_key_iv: "88a278b0ed6e08048c27151efa037ee0"
server_key_version: "encryptionkey@0"
I20260926 21:30:31.301222 5537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:30:31.304446 5551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:31.305697 5537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:31.305876 5537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "7041e794183d49379ac3886fc4345c28"
format_stamp: "Formatted at 2026-09-26 21:30:31 on dist-test-slave-87bt"
server_key: "d053dce0b931384886c969687eff4ef6"
server_key_iv: "88a278b0ed6e08048c27151efa037ee0"
server_key_version: "encryptionkey@0"
I20260926 21:30:31.306077 5537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:31.318702 5537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:31.319496 5537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:31.319765 5537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:31.337420 5537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36945
I20260926 21:30:31.337445 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36945 every 8 connection(s)
I20260926 21:30:31.338822 5537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:31.342491 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:31.346720 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5537
I20260926 21:30:31.346931 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:31.347702 388 external_mini_cluster.cc:1428] Setting key fa79f6ca931b1262ace3434254d564dc
I20260926 21:30:31.353660 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28: Bootstrap starting.
I20260926 21:30:31.356768 5603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:31.357815 5603 log.cc:824] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:31.361136 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28: No bootstrap required, opened a new log
I20260926 21:30:31.361598 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:31.349545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.0.97.62:36945)
0926 21:30:31.350473 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:31.350495 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:31.350538 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:30:31.352440 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:31.352462 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:31.352893 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:31.353275 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:31.353822 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:31.355204 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:31.356499 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:31.357362 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:31.357569 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:31.358890 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:31.358897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:31.358914 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:31.359611 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:31.359708 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:31.359733 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:31.359853 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:31.360046 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 21:30:31.360369 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":301,"threads_started":1}
I20260926 21:30:31.365226 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:30:31.369464 5603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041e794183d49379ac3886fc4345c28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } }
I20260926 21:30:31.369812 5603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:31.369932 5603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7041e794183d49379ac3886fc4345c28, State: Initialized, Role: FOLLOWER
I20260926 21:30:31.370553 5603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041e794183d49379ac3886fc4345c28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } }
I20260926 21:30:31.370774 5603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:31.370922 5603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:31.371101 5603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:31.373456 5603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041e794183d49379ac3886fc4345c28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } }
I20260926 21:30:31.373952 5603 leader_election.cc:302] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7041e794183d49379ac3886fc4345c28; no voters:
I20260926 21:30:31.374545 5603 leader_election.cc:288] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:31.374675 5608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:31.375454 5608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [term 1 LEADER]: Becoming Leader. State: Replica: 7041e794183d49379ac3886fc4345c28, State: Running, Role: LEADER
I20260926 21:30:31.375921 5608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041e794183d49379ac3886fc4345c28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } }
I20260926 21:30:31.376626 5603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:31.378772 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7041e794183d49379ac3886fc4345c28. Latest consensus state: current_term: 1 leader_uuid: "7041e794183d49379ac3886fc4345c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041e794183d49379ac3886fc4345c28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } } }
I20260926 21:30:31.379235 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:31.379487 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7041e794183d49379ac3886fc4345c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041e794183d49379ac3886fc4345c28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } } }
I20260926 21:30:31.379861 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:31.381605 5614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:31.385978 5614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:31.394080 5614 catalog_manager.cc:1347] Generated new cluster ID: 5152fa0cadb24b91acd43404b90c18ac
I20260926 21:30:31.394201 5614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:31.411882 5614 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:31.413530 5614 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:31.420379 5614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7041e794183d49379ac3886fc4345c28: Generated new TSK 0
I20260926 21:30:31.421227 5614 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 21:30:31.463032 5633 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 21:30:31.463214 5633 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 21:30:31.464161 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:31.435784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.0.97.62:36945)
0926 21:30:31.435980 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:31.435988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:31.436492 (+ 504us) server_negotiation.cc:407] Connection header received
0926 21:30:31.436834 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:31.436838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:31.436912 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:31.437063 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0926 21:30:31.437615 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:31.438347 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:31.439847 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:31.440285 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:31.440415 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:31.440956 (+ 541us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 21:30:31.463462 (+ 22506us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 21:30:31.463860 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 21:30:31.464293 5606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 21:30:31.465842 5632 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:31.435738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36945 (local address 127.0.0.1:38902)
0926 21:30:31.436370 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:31.436393 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:31.436598 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:31.437092 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:31.437097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:31.437111 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 21:30:31.437459 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:31.437468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:31.438574 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:31.438585 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:31.439700 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:31.439712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:31.440417 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:31.440487 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:31.440628 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:31.440686 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 21:30:31.463773 (+ 23087us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 21:30:31.464150 (+ 377us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":339,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:36945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 21:30:31.467742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5537
I20260926 21:30:31.468797 5598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:31.559507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5537
2026-09-26T21:30:31Z chronyd exiting
I20260926 21:30:31.609174 388 test_util.cc:175] -----------------------------------------------
I20260926 21:30:31.609308 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458187148965-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (704 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 21:30:31.612922 388 mini_oidc.cc:152] Starting JWKS server
I20260926 21:30:31.614770 388 webserver.cc:460] Webserver started at https://127.0.0.1:42123/ using document root <none> and password file <none>
I20260926 21:30:31.615031 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 21:30:31.615648 388 webserver.cc:460] Webserver started at http://127.0.0.1:43175/ using document root <none> and password file <none>
2026-09-26T21:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:31Z Disabled control of system clock
I20260926 21:30:31.641772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36743
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/testchainca.cert with env {}
W20260926 21:30:31.960723 5640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:31.961072 5640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:31.961222 5640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:31.968492 5640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:31.968642 5640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:31.968722 5640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:31.968834 5640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:31.968923 5640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 21:30:31.969025 5640 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 21:30:31.969105 5640 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42123/jwks.json
I20260926 21:30:31.979012 5640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:31.980409 5640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:31.981752 5640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:31.993925 5645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:31.994000 5646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:31.993963 5648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:31.994618 5640 server_base.cc:1034] running on GCE node
I20260926 21:30:31.995472 5640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:31.996929 5640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:31.998121 5640 hybrid_clock.cc:648] HybridClock initialized: now 1790458231998062 us; error 55 us; skew 500 ppm
I20260926 21:30:31.998533 5640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:32.006579 5640 webserver.cc:460] Webserver started at http://127.0.97.62:34479/ using document root <none> and password file <none>
I20260926 21:30:32.007302 5640 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:32.007433 5640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:32.007753 5640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:32.010634 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "f25323515c1746a494d2e8a4c2bc23a1"
format_stamp: "Formatted at 2026-09-26 21:30:32 on dist-test-slave-87bt"
server_key: "c83a6b36cf6fde6f68c3c4723437e883"
server_key_iv: "a1a309e7ff1927bf5880348628de89e3"
server_key_version: "encryptionkey@0"
I20260926 21:30:32.011432 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "f25323515c1746a494d2e8a4c2bc23a1"
format_stamp: "Formatted at 2026-09-26 21:30:32 on dist-test-slave-87bt"
server_key: "c83a6b36cf6fde6f68c3c4723437e883"
server_key_iv: "a1a309e7ff1927bf5880348628de89e3"
server_key_version: "encryptionkey@0"
I20260926 21:30:32.016355 5640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:30:32.019932 5654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:32.021320 5640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:30:32.021534 5640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "f25323515c1746a494d2e8a4c2bc23a1"
format_stamp: "Formatted at 2026-09-26 21:30:32 on dist-test-slave-87bt"
server_key: "c83a6b36cf6fde6f68c3c4723437e883"
server_key_iv: "a1a309e7ff1927bf5880348628de89e3"
server_key_version: "encryptionkey@0"
I20260926 21:30:32.021764 5640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:32.041244 5640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:32.042165 5640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:32.042543 5640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:32.061811 5640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36743
I20260926 21:30:32.061841 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36743 every 8 connection(s)
I20260926 21:30:32.063177 5640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:32.066923 5706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:32.070183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5640
I20260926 21:30:32.070401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:32.071031 388 external_mini_cluster.cc:1428] Setting key e210411ce545f44542e9ee581e1dc2a9
I20260926 21:30:32.078635 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1: Bootstrap starting.
I20260926 21:30:32.082048 5706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:32.083209 5706 log.cc:824] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:32.084251 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:32.072797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.0.97.62:36743)
0926 21:30:32.073726 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:32.073737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:32.073778 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:30:32.075344 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:32.075378 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:32.075852 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:32.076255 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:32.076832 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.078040 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.079856 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.080537 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.080752 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:32.081883 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:32.081890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:32.081903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:32.082529 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:32.082621 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:32.082641 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:32.082774 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:32.082994 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 21:30:32.083270 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":310,"threads_started":1}
I20260926 21:30:32.085940 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1: No bootstrap required, opened a new log
I20260926 21:30:32.087751 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:30:32.093648 5706 raft_consensus.cc:348] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25323515c1746a494d2e8a4c2bc23a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36743 } }
I20260926 21:30:32.094050 5706 raft_consensus.cc:374] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:32.094161 5706 raft_consensus.cc:729] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25323515c1746a494d2e8a4c2bc23a1, State: Initialized, Role: FOLLOWER
I20260926 21:30:32.094712 5706 consensus_queue.cc:260] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25323515c1746a494d2e8a4c2bc23a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36743 } }
I20260926 21:30:32.094919 5706 raft_consensus.cc:388] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:32.095027 5706 raft_consensus.cc:482] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:32.095148 5706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:32.097180 5706 raft_consensus.cc:504] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25323515c1746a494d2e8a4c2bc23a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36743 } }
I20260926 21:30:32.097641 5706 leader_election.cc:302] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f25323515c1746a494d2e8a4c2bc23a1; no voters:
I20260926 21:30:32.098197 5706 leader_election.cc:288] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:32.098368 5711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:32.099185 5711 raft_consensus.cc:686] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [term 1 LEADER]: Becoming Leader. State: Replica: f25323515c1746a494d2e8a4c2bc23a1, State: Running, Role: LEADER
I20260926 21:30:32.099743 5711 consensus_queue.cc:237] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25323515c1746a494d2e8a4c2bc23a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36743 } }
I20260926 21:30:32.100358 5706 sys_catalog.cc:564] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:32.102823 5713 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f25323515c1746a494d2e8a4c2bc23a1. Latest consensus state: current_term: 1 leader_uuid: "f25323515c1746a494d2e8a4c2bc23a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25323515c1746a494d2e8a4c2bc23a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36743 } } }
I20260926 21:30:32.103161 5713 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:32.104050 5719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:32.103900 5712 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f25323515c1746a494d2e8a4c2bc23a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25323515c1746a494d2e8a4c2bc23a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36743 } } }
I20260926 21:30:32.104206 5712 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:32.113852 5719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:32.122871 5719 catalog_manager.cc:1347] Generated new cluster ID: ae005130dfbe40158198706d18f69e37
I20260926 21:30:32.122998 5719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:32.132552 5719 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:32.133778 5719 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:32.140652 5719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f25323515c1746a494d2e8a4c2bc23a1: Generated new TSK 0
I20260926 21:30:32.141570 5719 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 21:30:32.181032 5736 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 21:30:32.181169 5736 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 21:30:32.182111 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:32.154824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.0.97.62:36743)
0926 21:30:32.155053 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:32.155062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:32.155525 (+ 463us) server_negotiation.cc:407] Connection header received
0926 21:30:32.155758 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:32.155762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:32.155826 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:32.155963 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0926 21:30:32.156528 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.157233 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.159277 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.159712 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.159814 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:32.160351 (+ 537us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 21:30:32.181553 (+ 21202us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 21:30:32.181886 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51128: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 21:30:32.182264 5709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51128: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 21:30:32.182551 5735 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:32.154798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36743 (local address 127.0.0.1:51128)
0926 21:30:32.155413 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:32.155433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:32.155584 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:32.156009 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:32.156016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:32.156032 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0926 21:30:32.156345 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:32.156358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.157370 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.157375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:32.159136 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:32.159151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.159845 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.159892 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:32.160091 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:32.160137 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 21:30:32.181821 (+ 21684us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 21:30:32.182116 (+ 295us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":321,"threads_started":1}
I20260926 21:30:32.183591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5640
I20260926 21:30:32.184787 5701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:32.273865 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5640
2026-09-26T21:30:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (713 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 21:30:32.326892 388 mini_oidc.cc:152] Starting JWKS server
I20260926 21:30:32.328753 388 webserver.cc:460] Webserver started at https://127.0.0.1:32785/ using document root <none> and password file <none>
I20260926 21:30:32.328976 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 21:30:32.329555 388 webserver.cc:460] Webserver started at http://127.0.0.1:45615/ using document root <none> and password file <none>
2026-09-26T21:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:32Z Disabled control of system clock
I20260926 21:30:32.355280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34309
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34309
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32785/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 21:30:32.689618 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:32.689982 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:32.690142 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:32.697429 5743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:32.697589 5743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:32.697686 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:32.697798 5743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:32.697875 5743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 21:30:32.697960 5743 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 21:30:32.698038 5743 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32785/jwks.json
I20260926 21:30:32.707368 5743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32785/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:32.708688 5743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:32.710592 5743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:32.722227 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:32.722301 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:32.722271 5751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:32.723191 5743 server_base.cc:1034] running on GCE node
I20260926 21:30:32.723937 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:32.725251 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:32.726527 5743 hybrid_clock.cc:648] HybridClock initialized: now 1790458232726469 us; error 38 us; skew 500 ppm
I20260926 21:30:32.726890 5743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:32.729082 5743 webserver.cc:460] Webserver started at http://127.0.97.62:36841/ using document root <none> and password file <none>
I20260926 21:30:32.729825 5743 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:32.729949 5743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:32.730285 5743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:32.733246 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "2d9db53d7354436c8b7878fa5c9f53f6"
format_stamp: "Formatted at 2026-09-26 21:30:32 on dist-test-slave-87bt"
server_key: "aa74a1f01728585487d874bbe7922995"
server_key_iv: "ce41817fb603fba8be65a22291625b44"
server_key_version: "encryptionkey@0"
I20260926 21:30:32.734025 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "2d9db53d7354436c8b7878fa5c9f53f6"
format_stamp: "Formatted at 2026-09-26 21:30:32 on dist-test-slave-87bt"
server_key: "aa74a1f01728585487d874bbe7922995"
server_key_iv: "ce41817fb603fba8be65a22291625b44"
server_key_version: "encryptionkey@0"
I20260926 21:30:32.738974 5743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:30:32.742331 5757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:32.743690 5743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:30:32.743886 5743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "2d9db53d7354436c8b7878fa5c9f53f6"
format_stamp: "Formatted at 2026-09-26 21:30:32 on dist-test-slave-87bt"
server_key: "aa74a1f01728585487d874bbe7922995"
server_key_iv: "ce41817fb603fba8be65a22291625b44"
server_key_version: "encryptionkey@0"
I20260926 21:30:32.744110 5743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:32.774153 5743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:32.775105 5743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:32.775405 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:32.793844 5743 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34309
I20260926 21:30:32.793866 5808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34309 every 8 connection(s)
I20260926 21:30:32.795261 5743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:32.798974 5809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:32.803656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5743
I20260926 21:30:32.803946 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:32.804903 388 external_mini_cluster.cc:1428] Setting key 805e8bda3d02727eadf25e91cdb803bf
I20260926 21:30:32.812611 5809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6: Bootstrap starting.
I20260926 21:30:32.816337 5809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:32.817595 5809 log.cc:824] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:32.820725 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:32.807003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.0.97.62:34309)
0926 21:30:32.808110 (+ 1107us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:32.808126 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:32.808180 (+ 54us) server_negotiation.cc:407] Connection header received
0926 21:30:32.810676 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:32.810708 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:32.811322 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:32.811755 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:32.812315 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.814263 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.815614 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.816682 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.816937 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:32.817958 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:32.817970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:32.817988 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:32.818743 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:32.818832 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:32.818852 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:32.818991 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:32.819179 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 21:30:32.819512 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":309,"threads_started":1}
I20260926 21:30:32.820942 5809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6: No bootstrap required, opened a new log
I20260926 21:30:32.824406 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:30:32.828720 5809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9db53d7354436c8b7878fa5c9f53f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34309 } }
I20260926 21:30:32.829097 5809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:32.829217 5809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9db53d7354436c8b7878fa5c9f53f6, State: Initialized, Role: FOLLOWER
I20260926 21:30:32.829844 5809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9db53d7354436c8b7878fa5c9f53f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34309 } }
I20260926 21:30:32.830083 5809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:32.830226 5809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:32.830475 5809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:32.833932 5809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9db53d7354436c8b7878fa5c9f53f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34309 } }
I20260926 21:30:32.834647 5809 leader_election.cc:302] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d9db53d7354436c8b7878fa5c9f53f6; no voters:
I20260926 21:30:32.839329 5809 leader_election.cc:288] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:32.839421 5814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:32.841075 5814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2d9db53d7354436c8b7878fa5c9f53f6, State: Running, Role: LEADER
I20260926 21:30:32.841636 5814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9db53d7354436c8b7878fa5c9f53f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34309 } }
I20260926 21:30:32.842276 5809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:32.844379 5815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d9db53d7354436c8b7878fa5c9f53f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9db53d7354436c8b7878fa5c9f53f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34309 } } }
I20260926 21:30:32.844755 5815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:32.846679 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d9db53d7354436c8b7878fa5c9f53f6. Latest consensus state: current_term: 1 leader_uuid: "2d9db53d7354436c8b7878fa5c9f53f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9db53d7354436c8b7878fa5c9f53f6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34309 } } }
I20260926 21:30:32.848101 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:32.851382 5819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:32.856353 5819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:32.865307 5819 catalog_manager.cc:1347] Generated new cluster ID: 223abeb12bf347fa91d6f748238b5464
I20260926 21:30:32.865430 5819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:32.879483 5819 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:32.880774 5819 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:32.894870 5819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d9db53d7354436c8b7878fa5c9f53f6: Generated new TSK 0
I20260926 21:30:32.895759 5819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:30:32.931219 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:32.922943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.0.97.62:34309)
0926 21:30:32.923128 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:32.923135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:32.923562 (+ 427us) server_negotiation.cc:407] Connection header received
0926 21:30:32.923753 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:32.923759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:32.923887 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:32.924046 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0926 21:30:32.924461 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.925578 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.926955 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.927641 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.927763 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:32.928210 (+ 447us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 21:30:32.930594 (+ 2384us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 21:30:32.930924 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 21:30:32.931345 5812 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 21:30:32.933053 5839 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:32.922828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34309 (local address 127.0.0.1:50262)
0926 21:30:32.923469 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:32.923484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:32.923595 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:32.924058 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:32.924061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:32.924070 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 21:30:32.924330 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:32.924337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.925708 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.925712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:32.926828 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:32.926839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:32.927754 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:32.927811 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:32.927927 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:32.927956 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 21:30:32.930785 (+ 2829us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 21:30:32.932656 (+ 1871us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":1577,"thread_start_us":342,"threads_started":1}
I20260926 21:30:32.934949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5743
I20260926 21:30:32.936071 5804 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 21:30:32.943079 5840 webserver.cc:533] Webserver: sslize failed:
E20260926 21:30:32.943203 5840 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 21:30:33.026125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5743
2026-09-26T21:30:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T21:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:33Z Disabled control of system clock
I20260926 21:30:33.106354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35505
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35505
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/testchainca.cert with env {}
W20260926 21:30:33.427197 5845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:33.427525 5845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:33.427744 5845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:33.435367 5845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:33.435491 5845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:33.435570 5845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:33.435683 5845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:33.435765 5845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 21:30:33.435874 5845 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 21:30:33.435935 5845 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 21:30:33.446152 5845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:33.447616 5845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:33.449014 5845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:33.460548 5850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:33.460634 5851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:33.460533 5853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:33.461720 5845 server_base.cc:1034] running on GCE node
I20260926 21:30:33.462664 5845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:33.464090 5845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:33.465289 5845 hybrid_clock.cc:648] HybridClock initialized: now 1790458233465227 us; error 54 us; skew 500 ppm
I20260926 21:30:33.465644 5845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:33.467911 5845 webserver.cc:460] Webserver started at http://127.0.97.62:46505/ using document root <none> and password file <none>
I20260926 21:30:33.468598 5845 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:33.468714 5845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:33.469058 5845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:33.471876 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "00ee78a646a740aca669088a048820b5"
format_stamp: "Formatted at 2026-09-26 21:30:33 on dist-test-slave-87bt"
server_key: "543e2b15dc0f05ce69d5cb055e2151ad"
server_key_iv: "39972b8400c8f9dfcd3e137c70f1035d"
server_key_version: "encryptionkey@0"
I20260926 21:30:33.472638 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "00ee78a646a740aca669088a048820b5"
format_stamp: "Formatted at 2026-09-26 21:30:33 on dist-test-slave-87bt"
server_key: "543e2b15dc0f05ce69d5cb055e2151ad"
server_key_iv: "39972b8400c8f9dfcd3e137c70f1035d"
server_key_version: "encryptionkey@0"
I20260926 21:30:33.477584 5845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:30:33.481014 5859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:33.482373 5845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:30:33.482606 5845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "00ee78a646a740aca669088a048820b5"
format_stamp: "Formatted at 2026-09-26 21:30:33 on dist-test-slave-87bt"
server_key: "543e2b15dc0f05ce69d5cb055e2151ad"
server_key_iv: "39972b8400c8f9dfcd3e137c70f1035d"
server_key_version: "encryptionkey@0"
I20260926 21:30:33.482836 5845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:33.509466 5845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:33.510267 5845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:33.510530 5845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:33.528775 5845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35505
I20260926 21:30:33.528795 5910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35505 every 8 connection(s)
I20260926 21:30:33.530133 5845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:33.533979 5911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:33.534403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5845
I20260926 21:30:33.534650 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:33.535305 388 external_mini_cluster.cc:1428] Setting key 7e14013ff6252fe443ffe12f740b7b87
I20260926 21:30:33.547794 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5: Bootstrap starting.
I20260926 21:30:33.549458 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:33.537403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.0.97.62:35505)
0926 21:30:33.538543 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:33.538554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:33.538603 (+ 49us) server_negotiation.cc:407] Connection header received
0926 21:30:33.540272 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:33.540296 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:33.540813 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:33.541177 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:33.541790 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:33.543119 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:33.545098 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:33.545812 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:33.545946 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:33.547180 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:33.547187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:33.547203 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:30:33.547740 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:33.547828 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:30:33.547854 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:33.547990 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:33.548223 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 21:30:33.548534 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":391,"threads_started":1}
I20260926 21:30:33.551321 5911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:33.552538 5911 log.cc:824] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:33.553396 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:30:33.555433 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5: No bootstrap required, opened a new log
I20260926 21:30:33.563359 5911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ee78a646a740aca669088a048820b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35505 } }
I20260926 21:30:33.563691 5911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:33.563771 5911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ee78a646a740aca669088a048820b5, State: Initialized, Role: FOLLOWER
I20260926 21:30:33.564251 5911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ee78a646a740aca669088a048820b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35505 } }
I20260926 21:30:33.564432 5911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:33.564527 5911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:33.564642 5911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:33.566978 5911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ee78a646a740aca669088a048820b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35505 } }
I20260926 21:30:33.567536 5911 leader_election.cc:302] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00ee78a646a740aca669088a048820b5; no voters:
I20260926 21:30:33.568099 5911 leader_election.cc:288] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:33.568284 5916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:33.568977 5916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [term 1 LEADER]: Becoming Leader. State: Replica: 00ee78a646a740aca669088a048820b5, State: Running, Role: LEADER
I20260926 21:30:33.569507 5916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ee78a646a740aca669088a048820b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35505 } }
I20260926 21:30:33.570241 5911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:33.572554 5917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00ee78a646a740aca669088a048820b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ee78a646a740aca669088a048820b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35505 } } }
I20260926 21:30:33.572999 5917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:33.573523 5918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00ee78a646a740aca669088a048820b5. Latest consensus state: current_term: 1 leader_uuid: "00ee78a646a740aca669088a048820b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ee78a646a740aca669088a048820b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35505 } } }
I20260926 21:30:33.573886 5918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:33.578997 5921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:33.584343 5921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:33.594667 5921 catalog_manager.cc:1347] Generated new cluster ID: f147ee9e72b64e7c9fa8ec12061f2aea
I20260926 21:30:33.594799 5921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:33.621275 5921 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:33.622666 5921 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:33.631062 5921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00ee78a646a740aca669088a048820b5: Generated new TSK 0
I20260926 21:30:33.632200 5921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:30:33.777827 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:33.655785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.0.97.62:35505)
0926 21:30:33.655974 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:33.655980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:33.656450 (+ 470us) server_negotiation.cc:407] Connection header received
0926 21:30:33.656650 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:33.656661 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:33.656766 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:33.656936 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0926 21:30:33.657372 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:33.658508 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:33.659897 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:33.660567 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:33.660690 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:33.661194 (+ 504us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 21:30:33.777160 (+115966us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 21:30:33.777531 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36436: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 21:30:33.777957 5914 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36436: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 21:30:33.778023 5940 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:33.655656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35505 (local address 127.0.0.1:36436)
0926 21:30:33.656353 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:33.656369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:33.656483 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:33.656937 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:33.656940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:33.656949 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 21:30:33.657226 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:33.657237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:33.658684 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:33.658688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:33.659751 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:33.659763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:33.660713 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:33.660761 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:33.660895 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:33.660930 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 21:30:33.777454 (+116524us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 21:30:33.777693 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35505: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":391,"threads_started":1}
I20260926 21:30:33.779294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5845
I20260926 21:30:33.780679 5906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:33.882174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5845
2026-09-26T21:30:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (835 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (405 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (414 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:34 dist-test-slave-87bt krb5kdc[5955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:34 dist-test-slave-87bt krb5kdc[5955](info): set up 2 sockets
Sep 26 21:30:34 dist-test-slave-87bt krb5kdc[5955](info): commencing operation
krb5kdc: starting...
W20260926 21:30:35.795398 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:35 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:36.014394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42761
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:36.334362 5968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:36.334748 5968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:36.334841 5968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:36.342192 5968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:36.342299 5968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:36.342345 5968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:36.342404 5968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:36.342515 5968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:36.352349 5968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:36.353585 5968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:36.354835 5968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:36.366528 5973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:36.366473 5976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:36.366466 5974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:36.367565 5968 server_base.cc:1034] running on GCE node
I20260926 21:30:36.368400 5968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:36.369802 5968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:36.371025 5968 hybrid_clock.cc:648] HybridClock initialized: now 1790458236370980 us; error 32 us; skew 500 ppm
I20260926 21:30:36.371348 5968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:36 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:36.374311 5968 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:30:36.375788 5968 webserver.cc:460] Webserver started at http://127.0.97.62:42905/ using document root <none> and password file <none>
I20260926 21:30:36.376437 5968 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:36.376542 5968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:36.376799 5968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:36.379755 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "03ccccfdd171436181062af82ad8f548"
format_stamp: "Formatted at 2026-09-26 21:30:36 on dist-test-slave-87bt"
server_key: "c33b8828fbb8c7937a007b31439e30d9"
server_key_iv: "dd2b1b75bd1fce4f3ffb9b05f1a80212"
server_key_version: "encryptionkey@0"
I20260926 21:30:36.380473 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "03ccccfdd171436181062af82ad8f548"
format_stamp: "Formatted at 2026-09-26 21:30:36 on dist-test-slave-87bt"
server_key: "c33b8828fbb8c7937a007b31439e30d9"
server_key_iv: "dd2b1b75bd1fce4f3ffb9b05f1a80212"
server_key_version: "encryptionkey@0"
I20260926 21:30:36.385669 5968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:30:36.389272 5983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:36.390748 5968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:36.390942 5968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "03ccccfdd171436181062af82ad8f548"
format_stamp: "Formatted at 2026-09-26 21:30:36 on dist-test-slave-87bt"
server_key: "c33b8828fbb8c7937a007b31439e30d9"
server_key_iv: "dd2b1b75bd1fce4f3ffb9b05f1a80212"
server_key_version: "encryptionkey@0"
I20260926 21:30:36.391144 5968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:36.406857 5968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:36.411193 5968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:36.411490 5968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:36.429366 5968 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42761
I20260926 21:30:36.429388 6034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42761 every 8 connection(s)
I20260926 21:30:36.430755 5968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:36.433606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 5968
I20260926 21:30:36.433797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:36.434413 388 external_mini_cluster.cc:1428] Setting key e911a202d192edb9502a511b69b41af3
I20260926 21:30:36.434546 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:36.448803 6035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548: Bootstrap starting.
Sep 26 21:30:36 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:36.452909 6035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:36.454277 6035 log.cc:824] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:36.457144 6035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548: No bootstrap required, opened a new log
I20260926 21:30:36.458287 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:36.436399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.0.97.62:42761)
0926 21:30:36.437219 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:36.437234 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:36.437287 (+ 53us) server_negotiation.cc:407] Connection header received
0926 21:30:36.439582 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:36.439613 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:36.440165 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:36.440644 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:36.441876 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:36.443863 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:36.445051 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:36.446164 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:36.446365 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:36.450265 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:36.450278 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:36.450311 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:36.450376 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:36.454287 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:36.455039 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:36.455047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:36.455057 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:36.455226 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:36.455607 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:36.455613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:36.455616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:36.456127 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:36.456408 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:36.456853 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 21:30:36.457322 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":314,"threads_started":1}
I20260926 21:30:36.464861 6035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ccccfdd171436181062af82ad8f548" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42761 } }
I20260926 21:30:36.465201 6035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:36.465293 6035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ccccfdd171436181062af82ad8f548, State: Initialized, Role: FOLLOWER
I20260926 21:30:36.465847 6035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ccccfdd171436181062af82ad8f548" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42761 } }
I20260926 21:30:36.466071 6035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:36.466221 6035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:36.466378 6035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:36.468703 6035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ccccfdd171436181062af82ad8f548" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42761 } }
I20260926 21:30:36.469180 6035 leader_election.cc:302] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03ccccfdd171436181062af82ad8f548; no voters:
I20260926 21:30:36.469740 6035 leader_election.cc:288] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:36.469902 6040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:36.470592 6040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [term 1 LEADER]: Becoming Leader. State: Replica: 03ccccfdd171436181062af82ad8f548, State: Running, Role: LEADER
I20260926 21:30:36.471114 6040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ccccfdd171436181062af82ad8f548" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42761 } }
I20260926 21:30:36.471788 6035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:36.474015 6042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03ccccfdd171436181062af82ad8f548. Latest consensus state: current_term: 1 leader_uuid: "03ccccfdd171436181062af82ad8f548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ccccfdd171436181062af82ad8f548" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42761 } } }
I20260926 21:30:36.474006 6041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03ccccfdd171436181062af82ad8f548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ccccfdd171436181062af82ad8f548" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42761 } } }
I20260926 21:30:36.474471 6041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:36.475073 6042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:36.475893 6045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:36.480728 6045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:36.489933 6045 catalog_manager.cc:1347] Generated new cluster ID: 055f5129fea849f78842f016ed1ffe80
I20260926 21:30:36.490064 6045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:36.499162 6045 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:36.500487 6045 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:36.517474 6045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03ccccfdd171436181062af82ad8f548: Generated new TSK 0
I20260926 21:30:36.518676 6045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:30:36.566262 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42761
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:36.899120 6061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:36.899397 6061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:36.899569 6061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:36.907012 6061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:36.907168 6061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:36.907378 6061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:36.917513 6061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:36.918952 6061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:36.920354 6061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:36.933928 6066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:36.933947 6067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:36.934010 6069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:36.934751 6061 server_base.cc:1034] running on GCE node
I20260926 21:30:36.935395 6061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:36.936367 6061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:36.937623 6061 hybrid_clock.cc:648] HybridClock initialized: now 1790458236937542 us; error 56 us; skew 500 ppm
I20260926 21:30:36.937964 6061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:36 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:36.941498 6061 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:30:36.943166 6061 webserver.cc:460] Webserver started at http://127.0.97.1:46565/ using document root <none> and password file <none>
I20260926 21:30:36.943835 6061 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:36.943935 6061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:36.944211 6061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:36.947160 6061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "1170c55fb83c4c44ae7a527c58dfd3d2"
format_stamp: "Formatted at 2026-09-26 21:30:36 on dist-test-slave-87bt"
server_key: "b8974328c5cfadfc30d27822a04510ca"
server_key_iv: "f297cbdf8d33b461886d7146b49c5550"
server_key_version: "encryptionkey@0"
I20260926 21:30:36.947880 6061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1170c55fb83c4c44ae7a527c58dfd3d2"
format_stamp: "Formatted at 2026-09-26 21:30:36 on dist-test-slave-87bt"
server_key: "b8974328c5cfadfc30d27822a04510ca"
server_key_iv: "f297cbdf8d33b461886d7146b49c5550"
server_key_version: "encryptionkey@0"
I20260926 21:30:36.953013 6061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:30:36.956547 6076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:36.957901 6061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:36.958115 6061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "1170c55fb83c4c44ae7a527c58dfd3d2"
format_stamp: "Formatted at 2026-09-26 21:30:36 on dist-test-slave-87bt"
server_key: "b8974328c5cfadfc30d27822a04510ca"
server_key_iv: "f297cbdf8d33b461886d7146b49c5550"
server_key_version: "encryptionkey@0"
I20260926 21:30:36.958331 6061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:36.992542 6061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:36.997037 6061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:36.997339 6061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:36.998348 6061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:36.999780 6061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:36.999889 6061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:37.000053 6061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:37.000128 6061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:37.033578 6061 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45827
I20260926 21:30:37.033638 6188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45827 every 8 connection(s)
I20260926 21:30:37.035048 6061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:37.037201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 6061
I20260926 21:30:37.037401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:37.038239 388 external_mini_cluster.cc:1428] Setting key 92bd6902efe587d61af852088a6f3ae0
Sep 26 21:30:37 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:30:37.061600 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:37.037887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52079 (local address 127.0.97.62:42761)
0926 21:30:37.038399 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:37.038405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:37.039752 (+ 1347us) server_negotiation.cc:407] Connection header received
0926 21:30:37.041751 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:37.041760 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:37.041851 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:37.041974 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:37.043638 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:37.044823 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:37.046549 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:37.046974 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:37.047132 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:37.055173 (+ 8041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:37.055180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:37.055183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:37.055220 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:37.058151 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:37.059069 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:37.059074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:37.059077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:37.059199 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:37.059705 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:37.059709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:37.059711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:37.059981 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:37.060149 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:37.060965 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 21:30:37.061212 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":252,"threads_started":1}
I20260926 21:30:37.062986 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:37.038603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42761 (local address 127.0.97.1:52079)
0926 21:30:37.039500 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:37.039551 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:37.041462 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:37.042194 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:37.042203 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:37.042863 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:37.043459 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:37.043477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:37.044995 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:37.044999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:37.046322 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:37.046334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:37.047247 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:37.047254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:37.047508 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:37.052084 (+ 4576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:37.052121 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:37.054856 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:37.058350 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:37.058365 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:37.058380 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:37.058904 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:37.059339 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:37.059345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:37.059349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:37.059547 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:37.060125 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:37.060134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:37.060562 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 21:30:37.060853 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 21:30:37.061483 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":344,"threads_started":1}
I20260926 21:30:37.064872 6189 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42761
I20260926 21:30:37.065198 6189 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:37.065959 6189 heartbeater.cc:511] Master 127.0.97.62:42761 requested a full tablet report, sending...
I20260926 21:30:37.068126 6000 ts_manager.cc:194] Registered new tserver with Master: 1170c55fb83c4c44ae7a527c58dfd3d2 (127.0.97.1:45827)
I20260926 21:30:37.069577 6000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:30:37.085659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42761
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:37.408318 6195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:37.408610 6195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:37.408821 6195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:37.416134 6195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:37.416285 6195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:37.416467 6195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:37.426263 6195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:37.427546 6195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:37.428956 6195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:37.442378 6201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:37.442493 6203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:37.442660 6200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:37.443166 6195 server_base.cc:1034] running on GCE node
I20260926 21:30:37.443766 6195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:37.446597 6195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:37.447865 6195 hybrid_clock.cc:648] HybridClock initialized: now 1790458237447779 us; error 100 us; skew 500 ppm
I20260926 21:30:37.448179 6195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:37 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:37.451571 6195 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:30:37.453102 6195 webserver.cc:460] Webserver started at http://127.0.97.2:32835/ using document root <none> and password file <none>
I20260926 21:30:37.453830 6195 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:37.453977 6195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:37.454291 6195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:37.457549 6195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "a93a564190514b978170e5e6b77a6276"
format_stamp: "Formatted at 2026-09-26 21:30:37 on dist-test-slave-87bt"
server_key: "04fb200a1a9b6f769d43c404d0ae701e"
server_key_iv: "2b176289ff575eae9765bbf66fbc6b5a"
server_key_version: "encryptionkey@0"
I20260926 21:30:37.458369 6195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a93a564190514b978170e5e6b77a6276"
format_stamp: "Formatted at 2026-09-26 21:30:37 on dist-test-slave-87bt"
server_key: "04fb200a1a9b6f769d43c404d0ae701e"
server_key_iv: "2b176289ff575eae9765bbf66fbc6b5a"
server_key_version: "encryptionkey@0"
I20260926 21:30:37.464021 6195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 21:30:37.467651 6210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:37.469053 6195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:30:37.469244 6195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "a93a564190514b978170e5e6b77a6276"
format_stamp: "Formatted at 2026-09-26 21:30:37 on dist-test-slave-87bt"
server_key: "04fb200a1a9b6f769d43c404d0ae701e"
server_key_iv: "2b176289ff575eae9765bbf66fbc6b5a"
server_key_version: "encryptionkey@0"
I20260926 21:30:37.469475 6195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:37.501641 6195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:37.505952 6195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:37.506256 6195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:37.507244 6195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:37.508601 6195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:37.508703 6195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:37.508876 6195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:37.508949 6195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:37.541810 6195 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38947
I20260926 21:30:37.541868 6322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38947 every 8 connection(s)
I20260926 21:30:37.543187 6195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:37.545768 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 6195
I20260926 21:30:37.545990 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:37.546397 388 external_mini_cluster.cc:1428] Setting key 2ed10a2030b1455cb769ee2efa845a34
Sep 26 21:30:37 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:37.564211 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:37.546630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45789 (local address 127.0.97.62:42761)
0926 21:30:37.546842 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:37.546849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:37.547666 (+ 817us) server_negotiation.cc:407] Connection header received
0926 21:30:37.549429 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:37.549434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:37.549516 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:37.549630 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:37.551215 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:37.551929 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:37.553675 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:37.554087 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:37.554186 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:37.558260 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:37.558265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:37.558268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:37.558299 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:37.560866 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:37.561732 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:37.561743 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:37.561747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:37.561867 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:37.562407 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:37.562413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:37.562415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:37.562715 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:37.562867 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:37.563650 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 21:30:37.563862 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:30:37.565153 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:37.546707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42761 (local address 127.0.97.2:45789)
0926 21:30:37.547504 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:37.547551 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:37.549169 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:37.549873 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:37.549882 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:37.550400 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:37.551049 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:37.551067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:37.552080 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:37.552088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:37.553500 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:37.553516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:37.554317 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:37.554327 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:37.554676 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:37.555397 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:37.555434 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:37.558024 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:37.561038 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:37.561049 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:37.561067 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:37.561597 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:37.561988 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:37.561994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:37.561998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:37.562284 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:37.562841 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:37.562849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:37.563249 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 21:30:37.563550 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 21:30:37.564087 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":371,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:30:37.566675 6323 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42761
I20260926 21:30:37.567004 6323 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:37.567803 6323 heartbeater.cc:511] Master 127.0.97.62:42761 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 21:30:37.569397 6000 ts_manager.cc:194] Registered new tserver with Master: a93a564190514b978170e5e6b77a6276 (127.0.97.2:38947)
I20260926 21:30:37.570209 6000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:30:37.593719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42761
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:37.914340 6328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:37.914688 6328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:37.914860 6328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:37.922178 6328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:37.922330 6328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:37.922535 6328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:37.932248 6328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:37.933573 6328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:37.935010 6328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:37.948433 6333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:37.948400 6336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:37.948341 6334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:37.949079 6328 server_base.cc:1034] running on GCE node
I20260926 21:30:37.949704 6328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:37.950759 6328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:37.951973 6328 hybrid_clock.cc:648] HybridClock initialized: now 1790458237951925 us; error 39 us; skew 500 ppm
I20260926 21:30:37.952316 6328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:37 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:37.955488 6328 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:30:37.956923 6328 webserver.cc:460] Webserver started at http://127.0.97.3:37665/ using document root <none> and password file <none>
I20260926 21:30:37.957631 6328 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:37.957779 6328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:37.958082 6328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:37.961076 6328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "84504c779eaf46588e62afd1d31294fe"
format_stamp: "Formatted at 2026-09-26 21:30:37 on dist-test-slave-87bt"
server_key: "592feae709278f71b7b88791c4ecd6a9"
server_key_iv: "a576fe95d6ba748107b3b509ad8784b0"
server_key_version: "encryptionkey@0"
I20260926 21:30:37.961897 6328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "84504c779eaf46588e62afd1d31294fe"
format_stamp: "Formatted at 2026-09-26 21:30:37 on dist-test-slave-87bt"
server_key: "592feae709278f71b7b88791c4ecd6a9"
server_key_iv: "a576fe95d6ba748107b3b509ad8784b0"
server_key_version: "encryptionkey@0"
I20260926 21:30:37.967346 6328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:30:37.970863 6343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:37.972429 6328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:30:37.972604 6328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "84504c779eaf46588e62afd1d31294fe"
format_stamp: "Formatted at 2026-09-26 21:30:37 on dist-test-slave-87bt"
server_key: "592feae709278f71b7b88791c4ecd6a9"
server_key_iv: "a576fe95d6ba748107b3b509ad8784b0"
server_key_version: "encryptionkey@0"
I20260926 21:30:37.972874 6328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:37.986076 6328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:37.990618 6328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:37.990936 6328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:37.991978 6328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:37.993314 6328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:37.993418 6328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:37.993561 6328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:37.993633 6328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:38.026399 6328 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45711
I20260926 21:30:38.026502 6455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45711 every 8 connection(s)
I20260926 21:30:38.027719 6328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:38.033010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 6328
I20260926 21:30:38.033180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:38.033708 388 external_mini_cluster.cc:1428] Setting key 7305c0cd230da55b9d92adbbeec6fc83
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:38.048544 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:38.030228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60481 (local address 127.0.97.62:42761)
0926 21:30:38.030468 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.030475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.031619 (+ 1144us) server_negotiation.cc:407] Connection header received
0926 21:30:38.033450 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.033455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.033544 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.033667 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.035561 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.036496 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.038336 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.038817 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.038959 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:38.042720 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:38.042727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:38.042732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:38.042784 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:38.045254 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.046025 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.046031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.046035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.046169 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.046680 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.046686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.046689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.046957 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:38.047110 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:38.047947 (+ 837us) server_negotiation.cc:299] Negotiation successful
0926 21:30:38.048138 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:30:38.049732 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:38.030759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42761 (local address 127.0.97.3:60481)
0926 21:30:38.031460 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:38.031499 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:38.033157 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:38.033930 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:38.033939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:38.034618 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:38.035389 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:38.035408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.036679 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.036687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:38.038156 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:38.038173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.039051 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.039059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:38.039300 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:38.039949 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:38.039982 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:38.042499 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:38.045417 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:38.045426 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:38.045445 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:38.045890 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:38.046287 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:38.046292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:38.046296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:38.046559 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:38.047086 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:38.047094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:38.047446 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 21:30:38.047756 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 21:30:38.048304 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":272,"threads_started":1}
I20260926 21:30:38.051049 6456 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42761
I20260926 21:30:38.051380 6456 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:38.052114 6456 heartbeater.cc:511] Master 127.0.97.62:42761 requested a full tablet report, sending...
I20260926 21:30:38.053485 6000 ts_manager.cc:194] Registered new tserver with Master: 84504c779eaf46588e62afd1d31294fe (127.0.97.3:45711)
I20260926 21:30:38.054250 6000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60481
I20260926 21:30:38.061607 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:30:38.072675 6189 heartbeater.cc:503] Master 127.0.97.62:42761 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/test-admin.keytab.
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:38.088822 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 21:30:38.093924 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:38.091466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.0.97.1:45827)
0926 21:30:38.092143 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.092150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.092176 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:30:38.092232 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.092238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.092449 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.092618 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.093498 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34654: BlockingRecv error: recv got EOF from 127.0.0.1:34654 (error 108)
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":307,"threads_started":1}
W20260926 21:30:38.094074 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:38.095762 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:38.097118 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:38.111716 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:38.098746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.0.97.1:45827)
0926 21:30:38.098952 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.098958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.099159 (+ 201us) server_negotiation.cc:407] Connection header received
0926 21:30:38.099302 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.099306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.099362 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.099445 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.100355 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.101281 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.102393 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.102991 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.103084 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:38.106552 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:38.106562 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:38.106577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:38.106606 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:38.109332 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.109936 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.109944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.109946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.110066 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.110373 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.110376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.110378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.110783 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:38.111003 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:38.111273 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 21:30:38.111456 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 21:30:38.117259 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:38.115638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.0.97.1:45827)
0926 21:30:38.115799 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.115803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.116118 (+ 315us) server_negotiation.cc:407] Connection header received
0926 21:30:38.116229 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.116232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.116285 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.116362 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.116996 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34666: BlockingRecv error: recv got EOF from 127.0.0.1:34666 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 21:30:38.117478 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:38.118990 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:38.120352 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:38.134686 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:38.121652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.0.97.1:45827)
0926 21:30:38.121883 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.121891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.122144 (+ 253us) server_negotiation.cc:407] Connection header received
0926 21:30:38.122338 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.122343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.122458 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.122579 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.123465 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.124159 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.125357 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.125809 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.125943 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:38.129955 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:38.129961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:38.129966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:38.130021 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:38.132532 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.133112 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.133115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.133117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.133235 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.133541 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.133544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.133546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.133841 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:38.134050 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:38.134220 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 21:30:38.134369 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 21:30:38.138705 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:38.137220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.0.97.1:45827)
0926 21:30:38.137427 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.137432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.137585 (+ 153us) server_negotiation.cc:407] Connection header received
0926 21:30:38.137698 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.137701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.137763 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.137836 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.138501 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34692: BlockingRecv error: recv got EOF from 127.0.0.1:34692 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 21:30:38.138864 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458187148965-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:38.140367 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:38.141636 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:38.156764 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:38.144157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.0.97.1:45827)
0926 21:30:38.144354 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.144360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.144508 (+ 148us) server_negotiation.cc:407] Connection header received
0926 21:30:38.144640 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.144644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.144702 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.144780 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.145761 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.146569 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.147736 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.148139 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.148230 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:38.151620 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:38.151625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:38.151628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:38.151659 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:38.154154 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.154830 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.154834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.154836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.154974 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.155367 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.155371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.155373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.155694 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:38.155965 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:38.156152 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 21:30:38.156369 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:30:38.158538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 6061
I20260926 21:30:38.169979 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:38.159275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.0.97.1:45827)
0926 21:30:38.159449 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.159456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.159799 (+ 343us) server_negotiation.cc:407] Connection header received
0926 21:30:38.159891 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.159895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.159951 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.160033 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.160888 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.161535 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.162883 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.163337 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.163426 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:38.165260 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:38.165265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:38.165268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:38.165293 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:38.167720 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.168295 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.168299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.168300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.168428 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.168727 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.168730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.168732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.168995 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:38.169173 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:38.169440 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 21:30:38.169598 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:30:38.170603 6184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:38.252640 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 6061
I20260926 21:30:38.268890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 6195
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:30:38.284855 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:38.269871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.0.97.2:38947)
0926 21:30:38.270671 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.270682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.270713 (+ 31us) server_negotiation.cc:407] Connection header received
0926 21:30:38.270814 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.270823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.271143 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.271328 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.272325 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.273433 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.274858 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.275408 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.275544 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:38.279442 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:38.279463 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:38.279476 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:38.279507 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:38.282018 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.282804 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.282813 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.282817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.282951 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.283347 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.283356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.283358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.283679 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:38.283978 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:38.284286 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 21:30:38.284542 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":394,"threads_started":1}
I20260926 21:30:38.286088 6318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:38.367441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 6195
I20260926 21:30:38.383596 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 6328
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:30:38.398470 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:38.384480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.0.97.3:45711)
0926 21:30:38.385079 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:38.385086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:38.385116 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:30:38.385175 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:38.385182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:38.385394 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:38.385525 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:38.386597 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.387606 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.388766 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:38.389306 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:38.389399 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:38.393124 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:38.393138 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:38.393152 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:38.393189 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:38.396150 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.396773 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.396780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.396782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.396915 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:38.397225 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:38.397228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:38.397230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:38.397542 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:38.397763 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:38.397958 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 21:30:38.398141 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":293,"threads_started":1}
I20260926 21:30:38.399735 6451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:38.480489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 6328
I20260926 21:30:38.496312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 5968
I20260926 21:30:38.497404 6030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:38.596942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 5968
2026-09-26T21:30:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[6483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[6483](info): set up 2 sockets
Sep 26 21:30:38 dist-test-slave-87bt krb5kdc[6483](info): commencing operation
krb5kdc: starting...
W20260926 21:30:39.683157 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:39 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 21:30:39.887851 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37433
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:40.202282 6496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:40.202739 6496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:40.202908 6496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:40.210930 6496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:40.211082 6496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:40.211190 6496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:40.211282 6496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:40.211349 6496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:40.221751 6496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:40.223187 6496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:40.224934 6496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:40.237329 6504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:40.237385 6502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:40.237397 6501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:40.243686 6496 server_base.cc:1034] running on GCE node
I20260926 21:30:40.244609 6496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:40.246090 6496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:40.247282 6496 hybrid_clock.cc:648] HybridClock initialized: now 1790458240247218 us; error 55 us; skew 500 ppm
I20260926 21:30:40.247612 6496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:40 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458240, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:40.250921 6496 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 21:30:40.252457 6496 webserver.cc:460] Webserver started at http://127.0.97.62:43319/ using document root <none> and password file <none>
I20260926 21:30:40.253196 6496 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:40.253343 6496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:40.253654 6496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:40.256783 6496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "05fc3954f5e743849b63c1bb2e6228ae"
format_stamp: "Formatted at 2026-09-26 21:30:40 on dist-test-slave-87bt"
server_key: "fe31abf779ea01556be046ee118ed940"
server_key_iv: "75df497e9b7eabe383e1f89ad821ff03"
server_key_version: "encryptionkey@0"
I20260926 21:30:40.257567 6496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "05fc3954f5e743849b63c1bb2e6228ae"
format_stamp: "Formatted at 2026-09-26 21:30:40 on dist-test-slave-87bt"
server_key: "fe31abf779ea01556be046ee118ed940"
server_key_iv: "75df497e9b7eabe383e1f89ad821ff03"
server_key_version: "encryptionkey@0"
I20260926 21:30:40.262801 6496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:30:40.266263 6511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:40.267629 6496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:40.267817 6496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "05fc3954f5e743849b63c1bb2e6228ae"
format_stamp: "Formatted at 2026-09-26 21:30:40 on dist-test-slave-87bt"
server_key: "fe31abf779ea01556be046ee118ed940"
server_key_iv: "75df497e9b7eabe383e1f89ad821ff03"
server_key_version: "encryptionkey@0"
I20260926 21:30:40.268013 6496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:40.304131 6496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:40.308231 6496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:40.308539 6496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:40.326647 6496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37433
I20260926 21:30:40.326666 6562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37433 every 8 connection(s)
I20260926 21:30:40.328074 6496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:40.331773 6563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:40.336927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 6496
I20260926 21:30:40.337170 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:40.337841 388 external_mini_cluster.cc:1428] Setting key d41b81dd53c02b7f41ca6cc43ba4f36a
I20260926 21:30:40.342475 6563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: Bootstrap starting.
I20260926 21:30:40.346102 6563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:40.347390 6563 log.cc:824] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:40.350147 6563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: No bootstrap required, opened a new log
Sep 26 21:30:40 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:40.358829 6563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } }
I20260926 21:30:40.359284 6563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:40.359392 6563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fc3954f5e743849b63c1bb2e6228ae, State: Initialized, Role: FOLLOWER
I20260926 21:30:40.360038 6563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } }
I20260926 21:30:40.360271 6563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:40.360455 6563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:40.360643 6563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:40.361196 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:40.340055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.97.62:37433)
0926 21:30:40.340878 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:40.340889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:40.340928 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:30:40.342656 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:40.342679 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:40.343072 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:40.343389 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:40.345231 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:40.346815 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:40.348489 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:40.349151 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:40.349358 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:40.353583 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:40.353590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:40.353604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:40.353641 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:40.356756 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:40.357560 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:40.357566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:40.357572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:40.357720 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:40.358107 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:40.358110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:40.358112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:40.358705 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:40.358911 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:40.359514 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 21:30:40.359943 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":410,"threads_started":1}
I20260926 21:30:40.363361 6563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } }
I20260926 21:30:40.363919 6563 leader_election.cc:302] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05fc3954f5e743849b63c1bb2e6228ae; no voters:
I20260926 21:30:40.364547 6563 leader_election.cc:288] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:40.364691 6568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:40.365394 6568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 1 LEADER]: Becoming Leader. State: Replica: 05fc3954f5e743849b63c1bb2e6228ae, State: Running, Role: LEADER
I20260926 21:30:40.365885 6568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } }
I20260926 21:30:40.366587 6563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:40.369093 6570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05fc3954f5e743849b63c1bb2e6228ae. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } }
I20260926 21:30:40.369525 6570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:40.371587 6569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } }
I20260926 21:30:40.371968 6569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:40.376963 6577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:40.381148 6577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:40.389629 6577 catalog_manager.cc:1347] Generated new cluster ID: b25f3348d34647c6b6f738f663807fdb
I20260926 21:30:40.389777 6577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:40.412353 6577 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:40.414300 6577 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:40.422262 6577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: Generated new TSK 0
I20260926 21:30:40.423508 6577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 21:30:40.470602 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37433
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:40.793721 6589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:40.793999 6589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:40.794091 6589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:40.801405 6589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:40.801535 6589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:40.801657 6589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:40.811576 6589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:40.812934 6589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:40.814241 6589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:40.827172 6594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:40.827275 6597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:40.827258 6595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:40.828311 6589 server_base.cc:1034] running on GCE node
I20260926 21:30:40.828960 6589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:40.829890 6589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:40.831096 6589 hybrid_clock.cc:648] HybridClock initialized: now 1790458240831051 us; error 50 us; skew 500 ppm
I20260926 21:30:40.831501 6589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:40 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458240, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:40.835942 6589 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 21:30:40.838137 6589 webserver.cc:460] Webserver started at http://127.0.97.1:39345/ using document root <none> and password file <none>
I20260926 21:30:40.838883 6589 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:40.839012 6589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:40.839300 6589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:40.842108 6589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "47732b6aa48c462cb58701cc6c40eb7d"
format_stamp: "Formatted at 2026-09-26 21:30:40 on dist-test-slave-87bt"
server_key: "20ffe65ff60de6290f349c38da038267"
server_key_iv: "48b7a97f32b0bcbcb9b8e20ad9e4def6"
server_key_version: "encryptionkey@0"
I20260926 21:30:40.842892 6589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "47732b6aa48c462cb58701cc6c40eb7d"
format_stamp: "Formatted at 2026-09-26 21:30:40 on dist-test-slave-87bt"
server_key: "20ffe65ff60de6290f349c38da038267"
server_key_iv: "48b7a97f32b0bcbcb9b8e20ad9e4def6"
server_key_version: "encryptionkey@0"
I20260926 21:30:40.848021 6589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 21:30:40.851568 6604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:40.852860 6589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:30:40.853044 6589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "47732b6aa48c462cb58701cc6c40eb7d"
format_stamp: "Formatted at 2026-09-26 21:30:40 on dist-test-slave-87bt"
server_key: "20ffe65ff60de6290f349c38da038267"
server_key_iv: "48b7a97f32b0bcbcb9b8e20ad9e4def6"
server_key_version: "encryptionkey@0"
I20260926 21:30:40.853266 6589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:40.880962 6589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:40.885267 6589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:40.885558 6589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:40.886546 6589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:40.887838 6589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:40.887941 6589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:40.888087 6589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:40.888159 6589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:40.920109 6589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35291
I20260926 21:30:40.920164 6716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35291 every 8 connection(s)
I20260926 21:30:40.921366 6589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:40.930240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 6589
I20260926 21:30:40.930490 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:40.931286 388 external_mini_cluster.cc:1428] Setting key 0ad5cc75dc27cc03251eb612f029a84d
Sep 26 21:30:40 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458240, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:40.941754 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:40.923886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42927 (local address 127.0.97.62:37433)
0926 21:30:40.924308 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:40.924313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:40.925124 (+ 811us) server_negotiation.cc:407] Connection header received
0926 21:30:40.927000 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:40.927006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:40.927097 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:40.927226 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:40.928694 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:40.929399 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:40.931838 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:40.932421 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:40.932542 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:40.936359 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:40.936365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:40.936368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:40.936397 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:40.938909 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:40.939595 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:40.939602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:40.939611 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:40.939719 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:40.940138 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:40.940142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:40.940147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:40.940393 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:40.940542 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:40.941205 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 21:30:40.941410 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":227,"threads_started":1}
I20260926 21:30:40.942759 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:40.924319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.1:42927)
0926 21:30:40.924971 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:40.925007 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:40.926733 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:40.927449 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:40.927457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:40.927961 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:40.928525 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:40.928541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:40.929551 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:40.929558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:40.931663 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:40.931677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:40.932658 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:40.932666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:40.932940 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:40.933716 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:40.933738 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:40.936145 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:40.939047 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:40.939058 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:40.939074 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:40.939470 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:40.939824 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:40.939829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:40.939833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:40.940014 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:40.940517 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:40.940525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:40.940847 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 21:30:40.941102 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 21:30:40.941596 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":262,"threads_started":1}
I20260926 21:30:40.944293 6717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37433
I20260926 21:30:40.944590 6717 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:40.945272 6717 heartbeater.cc:511] Master 127.0.97.62:37433 requested a full tablet report, sending...
I20260926 21:30:40.947540 6528 ts_manager.cc:194] Registered new tserver with Master: 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1:35291)
I20260926 21:30:40.949775 6528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42927
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 21:30:40.978221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37433
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:41.299082 6723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:41.299369 6723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:41.299461 6723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:41.306680 6723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:41.306800 6723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:41.306913 6723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:41.317358 6723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:41.318964 6723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:41.320345 6723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:41.333460 6728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:41.333505 6729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:41.334296 6731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:41.334386 6723 server_base.cc:1034] running on GCE node
I20260926 21:30:41.335150 6723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:41.336325 6723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:41.337558 6723 hybrid_clock.cc:648] HybridClock initialized: now 1790458241337493 us; error 51 us; skew 500 ppm
I20260926 21:30:41.337925 6723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:41 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458241, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:41.341629 6723 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 21:30:41.343226 6723 webserver.cc:460] Webserver started at http://127.0.97.2:34031/ using document root <none> and password file <none>
I20260926 21:30:41.343894 6723 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:41.344035 6723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:41.344329 6723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:41.347218 6723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "25878a5b4bad407f9592fdb2b6ce6d46"
format_stamp: "Formatted at 2026-09-26 21:30:41 on dist-test-slave-87bt"
server_key: "2e8ba5a2e7a629e7ea7864fe262591db"
server_key_iv: "e04cdc7ce58a423be49831ffd2083505"
server_key_version: "encryptionkey@0"
I20260926 21:30:41.347975 6723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "25878a5b4bad407f9592fdb2b6ce6d46"
format_stamp: "Formatted at 2026-09-26 21:30:41 on dist-test-slave-87bt"
server_key: "2e8ba5a2e7a629e7ea7864fe262591db"
server_key_iv: "e04cdc7ce58a423be49831ffd2083505"
server_key_version: "encryptionkey@0"
I20260926 21:30:41.353201 6723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:30:41.356578 6738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:41.358083 6723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:41.358255 6723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "25878a5b4bad407f9592fdb2b6ce6d46"
format_stamp: "Formatted at 2026-09-26 21:30:41 on dist-test-slave-87bt"
server_key: "2e8ba5a2e7a629e7ea7864fe262591db"
server_key_iv: "e04cdc7ce58a423be49831ffd2083505"
server_key_version: "encryptionkey@0"
I20260926 21:30:41.358467 6723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:41.370822 6723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:41.375149 6723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:41.375432 6723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:41.376336 6723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:41.377581 6723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:41.377688 6723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:41.377835 6723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:41.377904 6723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:41.411835 6723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35983
I20260926 21:30:41.411876 6850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35983 every 8 connection(s)
I20260926 21:30:41.413110 6723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:41.418237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 6723
I20260926 21:30:41.418413 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:41.418865 388 external_mini_cluster.cc:1428] Setting key 04a18f88cd8c03cdc0524ed40c0fbbf1
Sep 26 21:30:41 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458241, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:41.432696 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:41.415619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47321 (local address 127.0.97.62:37433)
0926 21:30:41.415794 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:41.415799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:41.417075 (+ 1276us) server_negotiation.cc:407] Connection header received
0926 21:30:41.419002 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:41.419007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:41.419099 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:41.419218 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:41.420930 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:41.421630 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:41.423378 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:41.423776 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:41.423908 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:41.427230 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:41.427238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:41.427247 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:41.427276 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:41.429746 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:41.430453 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:41.430459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:41.430462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:41.430593 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:41.431028 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:41.431045 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:41.431048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:41.431278 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:41.431430 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:41.432141 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 21:30:41.432359 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:30:41.433748 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:41.416082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.2:47321)
0926 21:30:41.416886 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:41.416922 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:41.418733 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:41.419449 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:41.419458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:41.420100 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:41.420759 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:41.420776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:41.421778 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:41.421790 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:41.423237 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:41.423249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:41.423992 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:41.423999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:41.424221 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:41.424889 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:41.424911 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:41.427018 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:41.429883 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:41.429891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:41.429907 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:41.430306 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:41.430718 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:41.430722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:41.430726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:41.430907 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:41.431404 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:41.431411 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:41.431750 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 21:30:41.432032 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 21:30:41.432555 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":375,"threads_started":1}
I20260926 21:30:41.435169 6851 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37433
I20260926 21:30:41.435511 6851 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:41.436411 6851 heartbeater.cc:511] Master 127.0.97.62:37433 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:30:41.438030 6528 ts_manager.cc:194] Registered new tserver with Master: 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2:35983)
I20260926 21:30:41.438952 6528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 21:30:41.464004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37433
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:41.784142 6856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:41.784411 6856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:41.784502 6856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:41.791680 6856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:41.791793 6856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:41.791903 6856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:41.801357 6856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:41.802693 6856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:41.803970 6856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:41.822037 6861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:41.821928 6862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:41.822008 6864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:41.823204 6856 server_base.cc:1034] running on GCE node
I20260926 21:30:41.823881 6856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:41.824904 6856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:41.826113 6856 hybrid_clock.cc:648] HybridClock initialized: now 1790458241826061 us; error 52 us; skew 500 ppm
I20260926 21:30:41.826579 6856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:41 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458241, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:41.829794 6856 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 21:30:41.831413 6856 webserver.cc:460] Webserver started at http://127.0.97.3:33293/ using document root <none> and password file <none>
I20260926 21:30:41.832118 6856 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:41.832248 6856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:41.832571 6856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:41.835565 6856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "36515a88910b4fd99cccc5f43c2ba661"
format_stamp: "Formatted at 2026-09-26 21:30:41 on dist-test-slave-87bt"
server_key: "d61d83436f88f80372102e53aa78380d"
server_key_iv: "e94bc946609a254691c42d27ec960b53"
server_key_version: "encryptionkey@0"
I20260926 21:30:41.836385 6856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "36515a88910b4fd99cccc5f43c2ba661"
format_stamp: "Formatted at 2026-09-26 21:30:41 on dist-test-slave-87bt"
server_key: "d61d83436f88f80372102e53aa78380d"
server_key_iv: "e94bc946609a254691c42d27ec960b53"
server_key_version: "encryptionkey@0"
I20260926 21:30:41.842623 6856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 21:30:41.846055 6871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:41.847450 6856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:41.847653 6856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "36515a88910b4fd99cccc5f43c2ba661"
format_stamp: "Formatted at 2026-09-26 21:30:41 on dist-test-slave-87bt"
server_key: "d61d83436f88f80372102e53aa78380d"
server_key_iv: "e94bc946609a254691c42d27ec960b53"
server_key_version: "encryptionkey@0"
I20260926 21:30:41.847869 6856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:41.899976 6856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:41.904138 6856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:41.904430 6856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:41.905421 6856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:41.906829 6856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:41.906934 6856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:41.907083 6856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:41.907156 6856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:41.939970 6856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36471
I20260926 21:30:41.940016 6983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36471 every 8 connection(s)
I20260926 21:30:41.941267 6856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:41.944039 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 6856
I20260926 21:30:41.944244 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:41.944810 388 external_mini_cluster.cc:1428] Setting key fc37a96945a2d229583a047980521227
I20260926 21:30:41.953019 6717 heartbeater.cc:503] Master 127.0.97.62:37433 was elected leader, sending a full tablet report...
Sep 26 21:30:41 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458241, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:41.962733 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:41.944057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46795 (local address 127.0.97.62:37433)
0926 21:30:41.944686 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:41.944693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:41.945799 (+ 1106us) server_negotiation.cc:407] Connection header received
0926 21:30:41.948099 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:41.948103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:41.948165 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:41.948246 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:41.949695 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:41.950506 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:41.952180 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:41.952643 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:41.952738 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:41.956558 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:41.956563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:41.956566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:41.956597 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:41.959921 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:41.960661 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:41.960665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:41.960668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:41.960782 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:41.961147 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:41.961150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:41.961152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:41.961403 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:41.961509 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:41.962223 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 21:30:41.962377 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":324,"threads_started":1}
I20260926 21:30:41.963779 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:41.944760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.3:46795)
0926 21:30:41.945611 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:41.945662 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:41.947766 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:41.948461 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:41.948469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:41.948973 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:41.949549 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:41.949567 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:41.950625 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:41.950628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:41.952037 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:41.952049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:41.952858 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:41.952863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:41.953054 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:41.953899 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:41.953927 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:41.956361 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:41.960077 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:41.960089 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:41.960109 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:41.960556 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:41.960884 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:41.960887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:41.960889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:41.961057 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:41.961535 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:41.961548 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:41.961863 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 21:30:41.962138 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 21:30:41.962652 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":359,"threads_started":1}
I20260926 21:30:41.965101 6984 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37433
I20260926 21:30:41.965415 6984 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:41.966126 6984 heartbeater.cc:511] Master 127.0.97.62:37433 requested a full tablet report, sending...
I20260926 21:30:41.967504 6528 ts_manager.cc:194] Registered new tserver with Master: 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471)
I20260926 21:30:41.968300 6528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46795
I20260926 21:30:41.972226 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260926 21:30:42.018294 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36515
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37433,127.0.97.61:36515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:42.343809 6990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:42.344168 6990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:42.344316 6990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:42.352080 6990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:42.352226 6990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:42.352329 6990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:42.352427 6990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:42.352500 6990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:42.363009 6990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37433,127.0.97.61:36515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
W20260926 21:30:42.363425 6990 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:37433,127.0.97.61:36515'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 21:30:42.364483 6990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:42.365792 6990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:42.378557 6998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:42.378484 6996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:42.378506 6995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:42.379602 6990 server_base.cc:1034] running on GCE node
I20260926 21:30:42.380415 6990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:42.381889 6990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:42.383132 6990 hybrid_clock.cc:648] HybridClock initialized: now 1790458242383065 us; error 76 us; skew 500 ppm
I20260926 21:30:42.383457 6990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:42 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458242, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:42.392583 6990 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 21:30:42.394073 6990 webserver.cc:460] Webserver started at http://127.0.97.61:43619/ using document root <none> and password file <none>
I20260926 21:30:42.394858 6990 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:42.394950 6990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:42.395215 6990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:42.398104 6990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data/instance:
uuid: "b638d68f18a54b75b5c9c83ed9be836e"
format_stamp: "Formatted at 2026-09-26 21:30:42 on dist-test-slave-87bt"
server_key: "91e9d61bee656827fab53e9c759cd085"
server_key_iv: "e4169d2e04757cb072d4eeaf9b6939fe"
server_key_version: "encryptionkey@0"
I20260926 21:30:42.398926 6990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal/instance:
uuid: "b638d68f18a54b75b5c9c83ed9be836e"
format_stamp: "Formatted at 2026-09-26 21:30:42 on dist-test-slave-87bt"
server_key: "91e9d61bee656827fab53e9c759cd085"
server_key_iv: "e4169d2e04757cb072d4eeaf9b6939fe"
server_key_version: "encryptionkey@0"
I20260926 21:30:42.404101 6990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:30:42.407545 7005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:42.408869 6990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:30:42.409057 6990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
uuid: "b638d68f18a54b75b5c9c83ed9be836e"
format_stamp: "Formatted at 2026-09-26 21:30:42 on dist-test-slave-87bt"
server_key: "91e9d61bee656827fab53e9c759cd085"
server_key_iv: "e4169d2e04757cb072d4eeaf9b6939fe"
server_key_version: "encryptionkey@0"
I20260926 21:30:42.409263 6990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:42.432282 6990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:42.436738 6990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:42.437009 6990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:42.441682 6851 heartbeater.cc:503] Master 127.0.97.62:37433 was elected leader, sending a full tablet report...
I20260926 21:30:42.455595 6990 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36515
I20260926 21:30:42.455627 7056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36515 every 8 connection(s)
I20260926 21:30:42.457026 6990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:30:42.458076 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 6990
I20260926 21:30:42.461423 7057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:42.474560 7057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:30:42 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 21:30:42.491513 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.460218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.0.97.61:36515)
0926 21:30:42.461120 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:42.461135 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:42.461542 (+ 407us) server_negotiation.cc:407] Connection header received
0926 21:30:42.464899 (+ 3357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:42.464928 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:42.465533 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:42.466060 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:42.470589 (+ 4529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.472866 (+ 2277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.475093 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.476355 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.476550 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.481968 (+ 5418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:42.481976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:42.481990 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:42.482025 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:42.486253 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.487137 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.487150 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.487162 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.487378 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.487908 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.487914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.487917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.488838 (+ 921us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:42.489129 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:42.489613 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 21:30:42.490079 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":369,"threads_started":1}
Sep 26 21:30:42 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458242, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:42.505074 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.478193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.0.97.62:37433)
0926 21:30:42.486846 (+ 8653us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:42.486853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:42.486873 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:30:42.486934 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:42.486938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:42.487040 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:42.487151 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:42.488970 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.490317 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.492469 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.493077 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.493227 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.497555 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:42.497564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:42.497571 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:42.497627 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:42.501538 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.502481 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.502489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.502493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.502646 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.503077 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.503084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.503089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.503567 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:42.503715 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:42.504345 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 21:30:42.504601 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8515,"thread_start_us":8382,"threads_started":1}
I20260926 21:30:42.505141 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.478141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:60410)
0926 21:30:42.479002 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:42.479058 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:42.479235 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:42.487386 (+ 8151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:42.487397 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:42.488338 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:42.488754 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.488770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.490486 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.490492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.492318 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.492334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.493264 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.493270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.493469 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.494939 (+ 1470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:42.494956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:42.497352 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:42.501708 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.501722 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.501730 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.502313 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.502769 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.502776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.502780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.502942 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.503714 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:42.503721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:42.503957 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 21:30:42.504215 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 21:30:42.504637 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":393,"threads_started":1}
I20260926 21:30:42.507205 7057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:30:42 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458242, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 21:30:42.520131 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.508123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.0.1:37422)
0926 21:30:42.508308 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:42.508326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:42.508431 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:42.508761 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:42.508764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:42.509074 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:42.509200 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.509207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.510790 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.510793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.511655 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.511665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.512662 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.512666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.512917 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.513520 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:42.513535 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:42.515483 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:42.518186 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.518189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.518192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.518624 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.518922 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.518925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.518927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.519049 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.519470 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:42.519473 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:42.519584 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 21:30:42.519684 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 21:30:42.519876 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":51}
I20260926 21:30:42.520298 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.508153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.0.97.61:36515)
0926 21:30:42.508314 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:42.508318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:42.508402 (+ 84us) server_negotiation.cc:407] Connection header received
0926 21:30:42.508546 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:42.508553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:42.508635 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:42.508748 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:42.509355 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.510674 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.511787 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.512517 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.512647 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.515644 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:42.515649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:42.515652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:42.515677 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:42.518077 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.518715 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.518719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.518721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.518831 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.519139 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.519142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.519144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.519379 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:42.519480 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:42.519760 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 21:30:42.519994 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":42}
I20260926 21:30:42.524648 7057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Bootstrap starting.
I20260926 21:30:42.527699 7057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:42.528671 7057 log.cc:824] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:42.531160 7057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: No bootstrap required, opened a new log
I20260926 21:30:42.538340 7057 raft_consensus.cc:348] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } }
I20260926 21:30:42.538749 7057 raft_consensus.cc:374] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:42.538856 7057 raft_consensus.cc:729] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b638d68f18a54b75b5c9c83ed9be836e, State: Initialized, Role: FOLLOWER
I20260926 21:30:42.539433 7057 consensus_queue.cc:260] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } }
I20260926 21:30:42.540220 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } } }
I20260926 21:30:42.540493 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:30:42.541193 7057 sys_catalog.cc:564] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:30:42.549481 7075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:30:42.549611 7075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:30:42.553032 6990 master_runner.cc:428] Detected that this master b638d68f18a54b75b5c9c83ed9be836e is joining an existing cluster
I20260926 21:30:42.553134 6990 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:36515
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260926 21:30:42.583456 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.557646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.0.97.62:37433)
0926 21:30:42.557873 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:42.557882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:42.562027 (+ 4145us) server_negotiation.cc:407] Connection header received
0926 21:30:42.566526 (+ 4499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:42.566536 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:42.566659 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:42.566750 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:42.568723 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.570163 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.572750 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.573347 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.573494 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.576336 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:42.576345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:42.576363 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:42.576407 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:42.580029 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.580866 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.580873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.580880 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.581031 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.581570 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.581576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.581580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.581964 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:42.582115 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:42.582798 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 21:30:42.583062 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:30:42.584905 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.557050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:60414)
0926 21:30:42.561893 (+ 4843us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:42.561917 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:42.566268 (+ 4351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:42.567040 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:42.567046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:42.567974 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:42.568508 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.568524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.570383 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.570391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.572586 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.572603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.573765 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.573772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.573989 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.574759 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:42.574787 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:42.576121 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:42.580164 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.580170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.580174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.580711 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.581165 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.581169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.581172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.581433 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.582191 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:42.582195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:42.582413 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 21:30:42.584014 (+ 1601us) client_negotiation.cc:240] Negotiation successful
0926 21:30:42.584402 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4676,"spinlock_wait_cycles":27520,"thread_start_us":4530,"threads_started":1}
I20260926 21:30:42.589368 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.557096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.0.97.61:36515)
0926 21:30:42.557312 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:42.557320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:42.571596 (+ 14276us) server_negotiation.cc:407] Connection header received
0926 21:30:42.571843 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:42.571852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:42.571915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:42.571988 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:42.573809 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.575311 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.577590 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.578570 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.578661 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.581153 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:42.581158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:42.581163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:42.581199 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:42.584761 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.585596 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.585600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.585603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.585771 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.586889 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.586893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.586896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.587231 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:42.587397 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:42.588776 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0926 21:30:42.588998 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 21:30:42.590308 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.557451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.0.1:37438)
0926 21:30:42.571500 (+ 14049us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:42.571520 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:42.571709 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:42.572173 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:42.572178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:42.572908 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:42.573647 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.573660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.575521 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.575527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.577450 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.577464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.578948 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.578955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.579128 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.579735 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:42.579757 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:42.580979 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:42.584926 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.584933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.584937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.585488 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.586533 (+ 1045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.586538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.586541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.586770 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.588314 (+ 1544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:42.588317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:42.588484 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 21:30:42.589600 (+ 1116us) client_negotiation.cc:240] Negotiation successful
0926 21:30:42.589892 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13901,"thread_start_us":13660,"threads_started":1}
Sep 26 21:30:42 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458240, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260926 21:30:42.605998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40023
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37433,127.0.97.61:36515,127.0.97.60:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:30:42.609150 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.592844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.0.97.61:36515)
0926 21:30:42.593173 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:42.593182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:42.595147 (+ 1965us) server_negotiation.cc:407] Connection header received
0926 21:30:42.595495 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:42.595500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:42.595585 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:42.595688 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:42.596747 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.597520 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.599044 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.599501 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.599632 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.603293 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:42.603299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:42.603302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:42.603360 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:42.606473 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.607181 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.607188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.607192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.607325 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.607689 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.607694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.607698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.607970 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:42.608121 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:42.608570 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 21:30:42.608776 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260926 21:30:42.609598 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.591837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.0.1:37442)
0926 21:30:42.595039 (+ 3202us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:42.595075 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:42.595268 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:42.595732 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:42.595740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:42.596401 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:42.596612 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.596620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.597655 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.597663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.598902 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.598913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.599670 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.599678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.599877 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.600549 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:42.600566 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:42.603093 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:42.606631 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.606643 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.606651 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.607054 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.607437 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.607442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.607446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.607577 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.608103 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:42.608111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:42.608292 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 21:30:42.608467 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 21:30:42.608776 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2978,"thread_start_us":2813,"threads_started":1}
I20260926 21:30:42.611506 6528 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:36515: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 36515
}
attrs {
promote: true
}
}
dest_uuid: "05fc3954f5e743849b63c1bb2e6228ae"
cas_config_opid_index: -1
I20260926 21:30:42.612600 6528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } }
I20260926 21:30:42.615720 7089 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b638d68f18a54b75b5c9c83ed9be836e (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } }
I20260926 21:30:42.616744 7032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:42.618796 7089 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:36515
I20260926 21:30:42.619485 6990 master.cc:537] Master@127.0.97.61:36515 shutting down...
I20260926 21:30:42.620350 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } }
I20260926 21:30:42.620774 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:42.622614 7032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Refusing update from remote peer 05fc3954f5e743849b63c1bb2e6228ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 21:30:42.623883 7088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:30:42.626646 6515 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 21:30:42.632517 6990 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:42.632906 6990 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 21:30:42.632907 6515 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae -> Peer b638d68f18a54b75b5c9c83ed9be836e (127.0.97.61:36515): Couldn't send request to peer b638d68f18a54b75b5c9c83ed9be836e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 21:30:42.633114 6990 tablet_replica.cc:330] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: stopping tablet replica
I20260926 21:30:42.639639 6990 master.cc:552] Master@127.0.97.61:36515 shutdown complete.
I20260926 21:30:42.639740 6990 master_runner.cc:305] Clearing existing system tablet
I20260926 21:30:42.640178 6990 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:42.640285 6990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:42.643383 6990 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:42.645130 7093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:42.645510 6990 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:42.645777 6990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:30:42.645854 6990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
uuid: "b638d68f18a54b75b5c9c83ed9be836e"
format_stamp: "Formatted at 2026-09-26 21:30:42 on dist-test-slave-87bt"
server_key: "91e9d61bee656827fab53e9c759cd085"
server_key_iv: "e4169d2e04757cb072d4eeaf9b6939fe"
server_key_version: "encryptionkey@0"
I20260926 21:30:42.650861 6990 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:42.657939 6990 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 21:30:42.658109 6990 log.cc:1196] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 21:30:42.658571 6990 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Deleting consensus metadata
I20260926 21:30:42.658831 6990 master_runner.cc:315] Copying system tablet from 127.0.97.62:37433
I20260926 21:30:42.666659 6990 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:37433
I20260926 21:30:42.685297 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.667609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.97.62:37433)
0926 21:30:42.667796 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:42.667802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:42.671017 (+ 3215us) server_negotiation.cc:407] Connection header received
0926 21:30:42.671192 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:42.671198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:42.671300 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:42.671414 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:42.672590 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.673970 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.675845 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.676687 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.676849 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.679060 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:42.679066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:42.679070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:42.679109 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:42.682412 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.683189 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.683194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.683197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.683340 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:42.683738 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:42.683742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:42.683745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:42.684102 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:42.684247 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:42.684724 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 21:30:42.684937 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:30:42.685331 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:42.667464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:60430)
0926 21:30:42.670852 (+ 3388us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:42.670883 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:42.671056 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:42.671489 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:42.671494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:42.672098 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:42.672373 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.672386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.674148 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.674154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.675708 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:42.675723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:42.676858 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:42.676864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:42.677047 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:42.677744 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:42.677768 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:42.678845 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:42.682569 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.682575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.682579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.683051 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.683460 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:42.683465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:42.683469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:42.683614 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:42.684223 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:42.684228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:42.684417 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 21:30:42.684566 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 21:30:42.684876 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3220,"thread_start_us":3123,"threads_started":1}
I20260926 21:30:42.686575 6548 tablet_copy_service.cc:140] P 05fc3954f5e743849b63c1bb2e6228ae: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b638d68f18a54b75b5c9c83ed9be836e ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60430)
I20260926 21:30:42.686833 6548 tablet_copy_service.cc:161] P 05fc3954f5e743849b63c1bb2e6228ae: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b638d68f18a54b75b5c9c83ed9be836e at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60430: session id = b638d68f18a54b75b5c9c83ed9be836e-00000000000000000000000000000000
I20260926 21:30:42.691324 6548 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 21:30:42.693503 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 21:30:42.700438 6990 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: tablet copy: Starting download of 0 data blocks...
I20260926 21:30:42.700779 6990 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: tablet copy: Starting download of 1 WAL segments...
I20260926 21:30:42.703550 6990 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 21:30:42.708065 6548 tablet_copy_service.cc:342] P 05fc3954f5e743849b63c1bb2e6228ae: Request end of tablet copy session b638d68f18a54b75b5c9c83ed9be836e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60430
I20260926 21:30:42.708283 6548 tablet_copy_service.cc:433] P 05fc3954f5e743849b63c1bb2e6228ae: ending tablet copy session b638d68f18a54b75b5c9c83ed9be836e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b638d68f18a54b75b5c9c83ed9be836e
I20260926 21:30:42.718644 6990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:42.721309 7100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:42.731596 7103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:42.734205 7101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:42.734771 6990 server_base.cc:1034] running on GCE node
I20260926 21:30:42.735112 6990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:42.735739 6990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:42.736954 6990 hybrid_clock.cc:648] HybridClock initialized: now 1790458242736888 us; error 35 us; skew 500 ppm
I20260926 21:30:42.737183 6990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:42 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458242, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:42.739053 6990 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 21:30:42.740278 6990 webserver.cc:460] Webserver started at http://127.0.97.61:36963/ using document root <none> and password file <none>
I20260926 21:30:42.740621 6990 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:42.740710 6990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:42.742990 6990 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:42.744604 7110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:42.745132 6990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 21:30:42.745282 6990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
uuid: "b638d68f18a54b75b5c9c83ed9be836e"
format_stamp: "Formatted at 2026-09-26 21:30:42 on dist-test-slave-87bt"
server_key: "91e9d61bee656827fab53e9c759cd085"
server_key_iv: "e4169d2e04757cb072d4eeaf9b6939fe"
server_key_version: "encryptionkey@0"
I20260926 21:30:42.745442 6990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:42.773065 6990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:42.776633 6990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:42.803555 6990 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36515
I20260926 21:30:42.803596 7161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36515 every 8 connection(s)
I20260926 21:30:42.804462 6990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:30:42.811450 7162 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:30:42.814028 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Bootstrap starting.
I20260926 21:30:42.826292 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:30:42.826786 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: Bootstrap complete.
I20260926 21:30:42.827625 7162 raft_consensus.cc:348] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } }
I20260926 21:30:42.827841 7162 raft_consensus.cc:729] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b638d68f18a54b75b5c9c83ed9be836e, State: Initialized, Role: LEARNER
I20260926 21:30:42.828151 7162 consensus_queue.cc:260] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } }
I20260926 21:30:42.828917 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } }
I20260926 21:30:42.829226 7162 sys_catalog.cc:564] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:42.829276 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: This master's current role is: LEARNER
I20260926 21:30:42.859232 7176 catalog_manager.cc:1259] Loaded cluster ID: b25f3348d34647c6b6f738f663807fdb
I20260926 21:30:42.859357 7176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: loading cluster ID for follower catalog manager: success
I20260926 21:30:42.861853 7176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: acquiring CA information for follower catalog manager: success
I20260926 21:30:42.863754 7176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:30:42.970690 6984 heartbeater.cc:503] Master 127.0.97.62:37433 was elected leader, sending a full tablet report...
W20260926 21:30:43.026518 7086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:43.026793 7086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:43.026957 7086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:43.034986 7086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:43.035107 7086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:43.035199 7086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:43.035305 7086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:43.035365 7086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:43.047324 7086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37433,127.0.97.61:36515,127.0.97.60:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:43.049055 7086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:43.050680 7086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 21:30:43.053668 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.042502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.0.1:37446)
0926 21:30:43.042682 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.042733 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.042889 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.046968 (+ 4079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.046973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.047003 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:43.047192 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.047203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.049126 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.049132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.051151 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.051167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.052873 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.052880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.053103 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.053172 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.053255 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.053396 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 21:30:43.054724 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.043053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.0.97.61:36515)
0926 21:30:43.046558 (+ 3505us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.046564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.046587 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:30:43.046660 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.046665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.046753 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.046895 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:43.047374 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.048946 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.051338 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.052667 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.053921 (+ 1254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.054167 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.054231 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.054361 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3328,"thread_start_us":3173,"threads_started":1}
I20260926 21:30:43.055907 7137 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 21:30:43.056504 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05fc3954f5e743849b63c1bb2e6228ae. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } }
I20260926 21:30:43.056797 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: This master's current role is: LEARNER
W20260926 21:30:43.064440 7185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:43.064399 7182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:43.064484 7183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:43.070758 7086 server_base.cc:1034] running on GCE node
I20260926 21:30:43.071678 7086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:43.073369 7086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:43.074529 7086 hybrid_clock.cc:648] HybridClock initialized: now 1790458243074483 us; error 38 us; skew 500 ppm
I20260926 21:30:43.074877 7086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:43.077914 7086 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 21:30:43.079531 7086 webserver.cc:460] Webserver started at http://127.0.97.60:42807/ using document root <none> and password file <none>
I20260926 21:30:43.080243 7086 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:43.080387 7086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:43.081076 7086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:43.084084 7086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data/instance:
uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd"
format_stamp: "Formatted at 2026-09-26 21:30:43 on dist-test-slave-87bt"
server_key: "059bcdeb42a8a4026cd9de08812f9d73"
server_key_iv: "0ef300dbf6de9fb09644617d92196853"
server_key_version: "encryptionkey@0"
I20260926 21:30:43.084817 7086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal/instance:
uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd"
format_stamp: "Formatted at 2026-09-26 21:30:43 on dist-test-slave-87bt"
server_key: "059bcdeb42a8a4026cd9de08812f9d73"
server_key_iv: "0ef300dbf6de9fb09644617d92196853"
server_key_version: "encryptionkey@0"
I20260926 21:30:43.089681 7086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 21:30:43.092902 7192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:43.094139 7086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:43.094316 7086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd"
format_stamp: "Formatted at 2026-09-26 21:30:43 on dist-test-slave-87bt"
server_key: "059bcdeb42a8a4026cd9de08812f9d73"
server_key_iv: "0ef300dbf6de9fb09644617d92196853"
server_key_version: "encryptionkey@0"
I20260926 21:30:43.094584 7086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:43.121356 7086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:43.125793 7086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:43.126073 7086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:43.145398 7086 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40023
I20260926 21:30:43.145423 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40023 every 8 connection(s)
I20260926 21:30:43.146927 7086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:30:43.150674 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:43.154155 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 7086
I20260926 21:30:43.159744 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:43.180868 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.163378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.0.97.62:37433)
0926 21:30:43.163597 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.163605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.164347 (+ 742us) server_negotiation.cc:407] Connection header received
0926 21:30:43.164493 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.164496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.164558 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.164640 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.165903 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.166975 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.169412 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.169874 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.169978 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.174760 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.174765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.174769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.174805 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.178109 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.178917 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.178920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.178923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.179042 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.179396 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.179399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.179401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.179736 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.179886 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.180444 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.180619 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:30:43.182282 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.163357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:60444)
0926 21:30:43.164182 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.164232 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.164378 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.164718 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.164725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.165529 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.165758 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.165767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.167103 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.167109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.169257 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.169268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.170126 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.170134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.170347 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.171075 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:43.171110 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:43.174577 (+ 3467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:43.178242 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.178253 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.178261 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.178802 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.179148 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.179152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.179155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.179297 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.179855 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:43.179860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:43.180088 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.180324 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.180898 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":70,"thread_start_us":397,"threads_started":1}
I20260926 21:30:43.182319 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.156200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.0.97.60:40023)
0926 21:30:43.156989 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.157004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.158932 (+ 1928us) server_negotiation.cc:407] Connection header received
0926 21:30:43.160691 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.160713 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.161148 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.161533 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.163085 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.165135 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.166850 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.168545 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.168760 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.174311 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.174323 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.174346 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.174407 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.177276 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.178131 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.178137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.178145 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.178380 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.178795 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.178800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.178803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.179521 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.179722 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.180246 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.180761 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":385,"threads_started":1}
I20260926 21:30:43.183974 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 21:30:43.197396 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.184620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.0.1:37458)
0926 21:30:43.184802 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.184822 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.184933 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.185750 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.185755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.186114 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.186250 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.186258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.187334 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.187346 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.188264 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.188282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.189076 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.189082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.189317 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.189971 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:43.189988 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:43.192042 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:43.195270 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.195274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.195277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.195694 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.196094 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.196097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.196100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.196225 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.196694 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:43.196697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:43.196817 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.196933 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.197156 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:30:43.197484 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.185252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.97.61:36515)
0926 21:30:43.185464 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.185470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.185490 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:30:43.185557 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.185561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.185618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.185731 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.186417 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.187180 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.188430 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.188855 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.189032 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.192243 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.192248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.192252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.192287 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.195118 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.195827 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.195832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.195834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.195987 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.196316 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.196319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.196321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.196599 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.196711 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.197017 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.197200 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 21:30:43.198542 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 21:30:43.211985 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.199415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40023 (local address 127.0.0.1:48180)
0926 21:30:43.199599 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.199619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.199725 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.200103 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.200107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.200448 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.200594 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.200602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.201586 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.201590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.202399 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.202410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.203253 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.203260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.203453 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.204134 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:43.204152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:43.206352 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:43.209582 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.209586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.209588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.209989 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.210537 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.210541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.210544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.210700 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.211267 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:43.211271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:43.211402 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.211507 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.211688 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 21:30:43.212081 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.199490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.0.97.60:40023)
0926 21:30:43.199647 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.199652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.199667 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:30:43.199877 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.199880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.199947 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.200033 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.200727 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.201446 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.202687 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.203089 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.203189 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.206587 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.206591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.206594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.206620 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.209453 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.210093 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.210096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.210098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.210195 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.210827 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.210833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.210835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.211133 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.211258 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.211662 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.211825 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:30:43.216815 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Bootstrap starting.
I20260926 21:30:43.219938 7244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:43.220978 7244 log.cc:824] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:43.223659 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: No bootstrap required, opened a new log
I20260926 21:30:43.231196 7244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } }
I20260926 21:30:43.231506 7244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:43.231638 7244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3840b9c4dd1a40f1ba8be8abc9cfcbcd, State: Initialized, Role: FOLLOWER
I20260926 21:30:43.232203 7244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } }
I20260926 21:30:43.233085 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } } }
I20260926 21:30:43.233402 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:30:43.234115 7244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:30:43.243793 7261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:30:43.243960 7261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:30:43.248158 7086 master_runner.cc:428] Detected that this master 3840b9c4dd1a40f1ba8be8abc9cfcbcd is joining an existing cluster
I20260926 21:30:43.248298 7086 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:40023
I20260926 21:30:43.275997 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.254157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.0.97.60:40023)
0926 21:30:43.254369 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.254377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.259665 (+ 5288us) server_negotiation.cc:407] Connection header received
0926 21:30:43.259839 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.259847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.259937 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.260049 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.262214 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.263397 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.265530 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.266296 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.266387 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.268915 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.268921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.268925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.268955 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.272798 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.273773 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.273779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.273782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.273920 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.274352 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.274357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.274360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.274714 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.274843 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.275414 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.275629 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61}
I20260926 21:30:43.276141 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.253533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.0.97.61:36515)
0926 21:30:43.253713 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.253721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.260083 (+ 6362us) server_negotiation.cc:407] Connection header received
0926 21:30:43.260319 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.260326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.260403 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.260486 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.262217 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.263276 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.265490 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.266133 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.266242 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.269064 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.269073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.269083 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.269127 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.272635 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.273689 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.273694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.273698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.273840 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.274399 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.274406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.274409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.274719 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.274835 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.275578 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.275798 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:30:43.276189 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.254214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.0.97.62:37433)
0926 21:30:43.254394 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.254400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.255596 (+ 1196us) server_negotiation.cc:407] Connection header received
0926 21:30:43.255805 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.255810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.255894 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.256003 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.261284 (+ 5281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.262529 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.264715 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.265330 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.265449 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.268480 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.268489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.268495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.268535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.272764 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.273689 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.273694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.273698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.273840 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.274343 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.274350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.274356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.274719 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.274839 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.275605 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.275798 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:30:43.277248 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.253654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:60448)
0926 21:30:43.255438 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.255469 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.255658 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.256244 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.256249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.257028 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.261025 (+ 3997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.261048 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.262737 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.262742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.264555 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.264568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.265736 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.265743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.265953 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.266745 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:43.266766 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:43.268260 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:43.272938 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.272942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.272945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.273550 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.273987 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.273991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.273995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.274192 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.274934 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:43.274938 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:43.275274 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.276400 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.276755 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1601,"mutex_wait_us":99,"spinlock_wait_cycles":10240,"thread_start_us":4525,"threads_started":1}
I20260926 21:30:43.277405 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.254064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.0.1:37464)
0926 21:30:43.259993 (+ 5929us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.260010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.260171 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.260646 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.260652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.261549 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.262054 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.262066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.263495 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.263500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.265344 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.265356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.266581 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.266588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.266782 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.267602 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:43.267628 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:43.268857 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:43.272846 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.272851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.272854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.273550 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.273987 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.273991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.273995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.274282 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.274934 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:43.274939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:43.275274 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.276386 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.276941 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5807,"mutex_wait_us":359,"spinlock_wait_cycles":38144,"thread_start_us":5177,"threads_started":1}
I20260926 21:30:43.277668 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.253708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40023 (local address 127.0.0.1:48184)
0926 21:30:43.259551 (+ 5843us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.259571 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.259700 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.260083 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.260088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.261538 (+ 1450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.262072 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.262084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.263633 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.263638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.265380 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.265394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.266707 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.266713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.266867 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.267433 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:43.267454 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:43.268742 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:43.272979 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.272984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.272988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.273657 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.274056 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.274061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.274064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.274239 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.274916 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:43.274920 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:43.275124 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.276760 (+ 1636us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.277103 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5708,"mutex_wait_us":241,"spinlock_wait_cycles":31488,"thread_start_us":1364,"threads_started":1}
I20260926 21:30:43.288537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 6589
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458240, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 21:30:43.298393 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.283261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40023 (local address 127.0.0.1:48190)
0926 21:30:43.283452 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.283467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.283548 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.283876 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.283879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.284220 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.284355 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.284362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.285749 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.285752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.286695 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.286706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.287714 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.287720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.287888 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.288486 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:43.288511 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:43.291845 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:43.295589 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.295595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.295600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.296093 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.296540 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.296546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.296550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.296699 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.297318 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:43.297323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:43.297549 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.297689 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.297953 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:30:43.299431 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.283348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.0.97.60:40023)
0926 21:30:43.283508 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.283514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.283532 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:43.283664 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.283669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.283749 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.283850 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.284521 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.285622 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.286818 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.287568 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.287705 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.292287 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.292293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.292298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.292362 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.295460 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.296242 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.296248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.296253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.296392 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.296838 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.296843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.296847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.297179 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.298617 (+ 1438us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.298740 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.298982 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 21:30:43.301685 6528 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:40023: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 40023
}
attrs {
promote: true
}
}
dest_uuid: "05fc3954f5e743849b63c1bb2e6228ae"
cas_config_opid_index: 5
I20260926 21:30:43.302770 6528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } }
I20260926 21:30:43.304415 7088 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3840b9c4dd1a40f1ba8be8abc9cfcbcd (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.309103 7137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 LEARNER]: Refusing update from remote peer 05fc3954f5e743849b63c1bb2e6228ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 21:30:43.310060 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.289401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.0.97.1:35291)
0926 21:30:43.289924 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.289930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.289955 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:30:43.290005 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.290011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.290202 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.290393 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.291648 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.293627 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.295471 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.296080 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.296217 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.301775 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.301787 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.301802 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.301839 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.305959 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.306842 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.306851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.306857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.307008 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.307453 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.307458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.307461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.307889 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.308190 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.309408 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.309668 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260926 21:30:43.310798 7219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:43.311483 6712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:43.316277 7219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Refusing update from remote peer 05fc3954f5e743849b63c1bb2e6228ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 21:30:43.328801 7274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260926 21:30:43.334121 7088 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:40023
I20260926 21:30:43.334923 7086 master.cc:537] Master@127.0.97.60:40023 shutting down...
I20260926 21:30:43.334975 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.336273 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:43.334137 7272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.006s
I20260926 21:30:43.351080 7137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3840b9c4dd1a40f1ba8be8abc9cfcbcd (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.355942 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: true } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.356215 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: This master's current role is: LEARNER
I20260926 21:30:43.356623 7271 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: attempting to promote NON_VOTER b638d68f18a54b75b5c9c83ed9be836e to VOTER
I20260926 21:30:43.357367 7271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } }
I20260926 21:30:43.359774 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 LEARNER]: Refusing update from remote peer 05fc3954f5e743849b63c1bb2e6228ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 21:30:43.360317 7274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 21:30:43.362767 7272 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b638d68f18a54b75b5c9c83ed9be836e (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.367898 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.368176 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:43.368445 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.368654 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:43.368635 7136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b638d68f18a54b75b5c9c83ed9be836e (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
W20260926 21:30:43.371330 6512 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae -> Peer 3840b9c4dd1a40f1ba8be8abc9cfcbcd (127.0.97.60:40023): Couldn't send request to peer 3840b9c4dd1a40f1ba8be8abc9cfcbcd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 21:30:43.372501 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.372730 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:30:43.378824 7250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:43.379107 7250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } }
W20260926 21:30:43.384027 7198 tablet.cc:2369] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:43.384550 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b638d68f18a54b75b5c9c83ed9be836e" is_pre_election: true
I20260926 21:30:43.384696 6538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fc3954f5e743849b63c1bb2e6228ae" is_pre_election: true
I20260926 21:30:43.384919 7250 leader_election.cc:288] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05fc3954f5e743849b63c1bb2e6228ae (127.0.97.62:37433), b638d68f18a54b75b5c9c83ed9be836e (127.0.97.61:36515)
I20260926 21:30:43.385715 7195 leader_election.cc:302] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3840b9c4dd1a40f1ba8be8abc9cfcbcd; no voters: 05fc3954f5e743849b63c1bb2e6228ae, b638d68f18a54b75b5c9c83ed9be836e
I20260926 21:30:43.386190 7250 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 21:30:43.386626 7086 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:43.387001 7086 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:43.387192 7086 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: stopping tablet replica
I20260926 21:30:43.392860 7086 master.cc:552] Master@127.0.97.60:40023 shutdown complete.
I20260926 21:30:43.392953 7086 master_runner.cc:305] Clearing existing system tablet
I20260926 21:30:43.393316 7086 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:43.393438 7086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:43.395607 7086 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:30:43.396965 7279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:43.397239 7086 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:43.397518 7086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:30:43.397622 7086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd"
format_stamp: "Formatted at 2026-09-26 21:30:43 on dist-test-slave-87bt"
server_key: "059bcdeb42a8a4026cd9de08812f9d73"
server_key_iv: "0ef300dbf6de9fb09644617d92196853"
server_key_version: "encryptionkey@0"
I20260926 21:30:43.400939 7086 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:43.405848 7086 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 21:30:43.406020 7086 log.cc:1196] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 21:30:43.406526 7086 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Deleting consensus metadata
I20260926 21:30:43.406762 7086 master_runner.cc:315] Copying system tablet from 127.0.97.62:37433
I20260926 21:30:43.409205 7086 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:37433
I20260926 21:30:43.423394 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.409964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.0.97.62:37433)
0926 21:30:43.410226 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.410236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.410960 (+ 724us) server_negotiation.cc:407] Connection header received
0926 21:30:43.411163 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.411168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.411243 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.411337 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.412454 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.413549 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.415216 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.415612 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.415753 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.417730 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:43.417736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:43.417741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:43.417780 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:43.420823 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.421477 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.421482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.421486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.421616 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:43.421998 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:43.422004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:43.422007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:43.422270 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:43.422432 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.422831 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.423037 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 21:30:43.423465 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.409945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:60460)
0926 21:30:43.410850 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.410865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.411024 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.411499 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.411504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.412022 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.412278 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.412289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.413729 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.413735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.415040 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.415051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.415937 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.415941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.416054 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.416770 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:43.416788 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:43.417544 (+ 756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:43.420973 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.420979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.420983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.421340 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.421731 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:43.421736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:43.421740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:43.421888 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:43.422387 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:43.422392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:43.422580 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.422725 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.423035 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":647,"threads_started":1}
I20260926 21:30:43.424300 6548 tablet_copy_service.cc:140] P 05fc3954f5e743849b63c1bb2e6228ae: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3840b9c4dd1a40f1ba8be8abc9cfcbcd ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60460)
I20260926 21:30:43.424474 6548 tablet_copy_service.cc:161] P 05fc3954f5e743849b63c1bb2e6228ae: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3840b9c4dd1a40f1ba8be8abc9cfcbcd at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60460: session id = 3840b9c4dd1a40f1ba8be8abc9cfcbcd-00000000000000000000000000000000
I20260926 21:30:43.426797 6548 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 21:30:43.428329 7086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 21:30:43.433398 7086 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: tablet copy: Starting download of 0 data blocks...
I20260926 21:30:43.433698 7086 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: tablet copy: Starting download of 1 WAL segments...
I20260926 21:30:43.436520 7086 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 21:30:43.440006 6548 tablet_copy_service.cc:342] P 05fc3954f5e743849b63c1bb2e6228ae: Request end of tablet copy session 3840b9c4dd1a40f1ba8be8abc9cfcbcd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60460
I20260926 21:30:43.440146 6548 tablet_copy_service.cc:433] P 05fc3954f5e743849b63c1bb2e6228ae: ending tablet copy session 3840b9c4dd1a40f1ba8be8abc9cfcbcd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3840b9c4dd1a40f1ba8be8abc9cfcbcd
I20260926 21:30:43.452437 7086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:43.455284 7286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:43.455621 7287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:43.466706 7086 server_base.cc:1034] running on GCE node
W20260926 21:30:43.468011 7289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:43.468544 7086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:43.469257 7086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:43.470506 7086 hybrid_clock.cc:648] HybridClock initialized: now 1790458243470461 us; error 35 us; skew 500 ppm
I20260926 21:30:43.470827 7086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:30:43.471432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 6589
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:43.473464 7086 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 21:30:43.483731 7086 webserver.cc:460] Webserver started at http://127.0.97.60:33129/ using document root <none> and password file <none>
I20260926 21:30:43.484153 7086 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:43.484313 7086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:43.487565 7086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 21:30:43.489660 7296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:43.489675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35291
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37433
--webserver_port=39345
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515,127.0.97.60:40023 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:30:43.490334 7086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:43.490477 7086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd"
format_stamp: "Formatted at 2026-09-26 21:30:43 on dist-test-slave-87bt"
server_key: "059bcdeb42a8a4026cd9de08812f9d73"
server_key_iv: "0ef300dbf6de9fb09644617d92196853"
server_key_version: "encryptionkey@0"
I20260926 21:30:43.490656 7086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:43.514884 7086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:43.518857 7086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:43.541832 7086 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40023
I20260926 21:30:43.541986 7348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40023 every 8 connection(s)
I20260926 21:30:43.542891 7086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:30:43.548398 7349 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:30:43.552102 7349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Bootstrap starting.
I20260926 21:30:43.569599 7349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:30:43.570101 7349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Bootstrap complete.
I20260926 21:30:43.571138 7349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } }
I20260926 21:30:43.571420 7349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3840b9c4dd1a40f1ba8be8abc9cfcbcd, State: Initialized, Role: LEARNER
I20260926 21:30:43.571702 7349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } }
I20260926 21:30:43.572790 7349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:43.572548 7352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.573127 7352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: This master's current role is: LEARNER
I20260926 21:30:43.580992 7363 catalog_manager.cc:1259] Loaded cluster ID: b25f3348d34647c6b6f738f663807fdb
I20260926 21:30:43.581122 7363 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: loading cluster ID for follower catalog manager: success
I20260926 21:30:43.583490 7363 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: acquiring CA information for follower catalog manager: success
I20260926 21:30:43.585335 7363 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:30:43.821645 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.814046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40023 (local address 127.0.0.1:48204)
0926 21:30:43.814643 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.814666 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.814805 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.815696 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.815699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.815721 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:43.815918 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.815929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.817770 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.817774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.819079 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.819091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.821088 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.821093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.821219 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.821243 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 21:30:43.821311 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 21:30:43.821412 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":252,"threads_started":1}
I20260926 21:30:43.821826 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.814169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.0.97.60:40023)
0926 21:30:43.815302 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.815309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.815339 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:30:43.815417 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.815424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.815527 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.815670 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:43.816090 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.817616 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.819504 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.820900 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.821040 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:43.821283 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:43.821337 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 21:30:43.821471 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":726,"threads_started":1}
I20260926 21:30:43.822693 7324 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 21:30:43.823211 7352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05fc3954f5e743849b63c1bb2e6228ae. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:43.823500 7352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: This master's current role is: LEARNER
W20260926 21:30:43.845474 7299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:43.845855 7299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:43.846052 7299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:43.854884 7299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:43.854987 7299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:43.855185 7299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:43.865692 7299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39345
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515,127.0.97.60:40023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:43.867378 7299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:43.868841 7299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:43.882025 7370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:43.882048 7373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:43.882089 7371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:43.883172 7299 server_base.cc:1034] running on GCE node
I20260926 21:30:43.883718 7299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:43.884569 7299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:43.885759 7299 hybrid_clock.cc:648] HybridClock initialized: now 1790458243885722 us; error 28 us; skew 500 ppm
I20260926 21:30:43.886029 7299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:43.889488 7299 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 21:30:43.891104 7299 webserver.cc:460] Webserver started at http://127.0.97.1:39345/ using document root <none> and password file <none>
I20260926 21:30:43.891765 7299 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:43.891916 7299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:43.897004 7299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 21:30:43.899648 7380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:43.900949 7299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:43.901120 7299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "47732b6aa48c462cb58701cc6c40eb7d"
format_stamp: "Formatted at 2026-09-26 21:30:40 on dist-test-slave-87bt"
server_key: "20ffe65ff60de6290f349c38da038267"
server_key_iv: "48b7a97f32b0bcbcb9b8e20ad9e4def6"
server_key_version: "encryptionkey@0"
I20260926 21:30:43.902073 7299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:43.931533 7299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:43.935953 7299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:43.936223 7299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:43.937278 7299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:43.938637 7299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:43.938750 7299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:43.938879 7299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:43.938975 7299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:43.974287 7299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35291
I20260926 21:30:43.974393 7492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35291 every 8 connection(s)
I20260926 21:30:43.976109 7299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:43.981585 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 7299
I20260926 21:30:43.981997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 6723
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 21:30:43 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458243, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 21:30:44 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 21:30:44.013309 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.979420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36101 (local address 127.0.97.60:40023)
0926 21:30:43.979940 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.979950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.992141 (+ 12191us) server_negotiation.cc:407] Connection header received
0926 21:30:43.992340 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.992345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.992429 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.992520 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.994559 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.995663 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.998711 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.999343 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.999451 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.005807 (+ 6356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:44.005814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:44.005821 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:44.005867 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:44.009381 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.010234 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.010240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.010245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.010399 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.011337 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.011342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.011346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.011740 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:44.011866 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:44.012739 (+ 873us) server_negotiation.cc:299] Negotiation successful
0926 21:30:44.012975 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260926 21:30:44.013594 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.979315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34483 (local address 127.0.97.61:36515)
0926 21:30:43.991695 (+ 12380us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.991702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.997270 (+ 5568us) server_negotiation.cc:407] Connection header received
0926 21:30:43.997416 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.997421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.997514 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.997618 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.998995 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.000003 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.002615 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.003125 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.003259 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.006309 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:44.006314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:44.006318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:44.006350 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:44.010149 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.011034 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.011040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.011045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.011186 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.011693 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.011700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.011706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.012068 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:44.012206 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:44.012970 (+ 764us) server_negotiation.cc:299] Negotiation successful
0926 21:30:44.013182 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12248,"thread_start_us":12397,"threads_started":1}
I20260926 21:30:44.015125 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.984980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.97.1:34483)
0926 21:30:43.997145 (+ 12165us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.997163 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.997309 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.997799 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.997803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.998444 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.998797 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.998808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.000165 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.000174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.002409 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:44.002451 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.003316 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.003332 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.003501 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.004015 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:44.004031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:44.006124 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:44.010307 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.010312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.010316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.010891 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.011313 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.011317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.011321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.011492 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.012265 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:44.012269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:44.012413 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 21:30:44.012627 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 21:30:44.014063 (+ 1436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12041,"mutex_wait_us":959,"spinlock_wait_cycles":24960,"thread_start_us":6761,"threads_started":1}
I20260926 21:30:44.015939 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.979621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46291 (local address 127.0.97.62:37433)
0926 21:30:43.990938 (+ 11317us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.990948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.990977 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:30:43.991059 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.991067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.991174 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.991326 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.994583 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.995684 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.999479 (+ 3795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.000097 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.000233 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.007365 (+ 7132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:44.007371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:44.007377 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:44.007424 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:44.011315 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.013728 (+ 2413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.013734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.013737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.013901 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.014365 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.014371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.014374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.014735 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:44.014891 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:44.015305 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 21:30:44.015526 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11146,"thread_start_us":10953,"threads_started":1}
I20260926 21:30:44.016451 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.980425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.1:46291)
0926 21:30:43.985265 (+ 4840us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.985335 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.988402 (+ 3067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.991691 (+ 3289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.991709 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.992708 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.994308 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.994349 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.995879 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.995887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.998804 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.998814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.000495 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.000500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.000644 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.001228 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:44.001249 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:44.007130 (+ 5881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:44.011611 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.011615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.011619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.012055 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.014023 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.014028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.014032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.014185 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.014869 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:44.014874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:44.015018 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 21:30:44.015173 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 21:30:44.015781 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4469,"mutex_wait_us":715,"thread_start_us":16259,"threads_started":1}
I20260926 21:30:44.016752 7495 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36515
I20260926 21:30:44.017091 7495 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:44.017211 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.984418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40023 (local address 127.0.97.1:36101)
0926 21:30:43.992028 (+ 7610us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:43.992045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:43.992202 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:43.992715 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:43.992721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:43.993375 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:43.994306 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.994337 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.995866 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.995871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:43.998540 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:43.998553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.999753 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:43.999759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.000035 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.000873 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:44.000910 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:44.005519 (+ 4609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:44.009514 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.009524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.009542 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.010103 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.010563 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.010569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.010573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.010811 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.011956 (+ 1145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:44.011965 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:44.012355 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 21:30:44.012641 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 21:30:44.013196 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7501,"mutex_wait_us":162,"thread_start_us":1469,"threads_started":1}
I20260926 21:30:44.017673 7494 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37433
I20260926 21:30:44.017803 7494 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:44.018038 7495 heartbeater.cc:511] Master 127.0.97.61:36515 requested a full tablet report, sending...
I20260926 21:30:44.018095 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:43.983573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.0.97.2:35983)
0926 21:30:43.992523 (+ 8950us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:43.992532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:43.993302 (+ 770us) server_negotiation.cc:407] Connection header received
0926 21:30:43.993378 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:43.993386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:43.993657 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:43.994207 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:43.997745 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:43.999296 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.001168 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.002021 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.002141 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.008547 (+ 6406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:44.008564 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:44.008583 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:44.008623 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:44.014174 (+ 5551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.015347 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.015355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.015358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.015503 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.015939 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.015943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.015946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.016296 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:44.016590 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:44.017429 (+ 839us) server_negotiation.cc:299] Negotiation successful
0926 21:30:44.017681 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8755,"thread_start_us":7040,"threads_started":1}
I20260926 21:30:44.018133 7493 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40023
I20260926 21:30:44.018254 7493 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:44.018399 7494 heartbeater.cc:511] Master 127.0.97.62:37433 requested a full tablet report, sending...
I20260926 21:30:44.019125 7493 heartbeater.cc:511] Master 127.0.97.60:40023 requested a full tablet report, sending...
I20260926 21:30:44.019404 6846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:44.020542 7127 ts_manager.cc:194] Registered new tserver with Master: 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1:35291)
I20260926 21:30:44.020632 7314 ts_manager.cc:194] Registered new tserver with Master: 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1:35291)
I20260926 21:30:44.021788 6528 ts_manager.cc:194] Re-registered known tserver with Master: 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1:35291)
I20260926 21:30:44.022970 6528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46291
I20260926 21:30:44.104467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 6723
I20260926 21:30:44.120625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35983
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37433
--webserver_port=34031
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515,127.0.97.60:40023 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:30:44.209389 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: true } } }
I20260926 21:30:44.209741 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:44.211614 7507 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: attempting to promote NON_VOTER 3840b9c4dd1a40f1ba8be8abc9cfcbcd to VOTER
I20260926 21:30:44.212250 7507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } }
I20260926 21:30:44.214241 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Refusing update from remote peer 05fc3954f5e743849b63c1bb2e6228ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 21:30:44.214243 7324 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 LEARNER]: Refusing update from remote peer 05fc3954f5e743849b63c1bb2e6228ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 21:30:44.214907 7506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 21:30:44.215307 7507 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 21:30:44.217998 7506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3840b9c4dd1a40f1ba8be8abc9cfcbcd (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:44.218811 7136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3840b9c4dd1a40f1ba8be8abc9cfcbcd (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:44.222242 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:44.222584 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:44.222909 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:44.223167 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:44.226928 7510 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:44.227375 7510 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:30:44.229373 7506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:44.229627 7506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:44.230072 7324 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3840b9c4dd1a40f1ba8be8abc9cfcbcd (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:44.233008 7352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05fc3954f5e743849b63c1bb2e6228ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:44.233315 7352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: This master's current role is: FOLLOWER
W20260926 21:30:44.461704 7505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:44.462023 7505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:44.462181 7505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:44.469551 7505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:44.469718 7505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:44.469887 7505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:44.480333 7505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34031
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515,127.0.97.60:40023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:44.481734 7505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:44.483152 7505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:44.496520 7515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:44.496567 7516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:44.496534 7518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:44.497412 7505 server_base.cc:1034] running on GCE node
I20260926 21:30:44.498060 7505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:44.499091 7505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:44.500337 7505 hybrid_clock.cc:648] HybridClock initialized: now 1790458244500286 us; error 63 us; skew 500 ppm
I20260926 21:30:44.500663 7505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:44 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458244, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:44.503847 7505 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 21:30:44.505401 7505 webserver.cc:460] Webserver started at http://127.0.97.2:34031/ using document root <none> and password file <none>
I20260926 21:30:44.506187 7505 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:44.506343 7505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:44.511668 7505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:30:44.514623 7525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:44.515975 7505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:44.516163 7505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "25878a5b4bad407f9592fdb2b6ce6d46"
format_stamp: "Formatted at 2026-09-26 21:30:41 on dist-test-slave-87bt"
server_key: "2e8ba5a2e7a629e7ea7864fe262591db"
server_key_iv: "e04cdc7ce58a423be49831ffd2083505"
server_key_version: "encryptionkey@0"
I20260926 21:30:44.517231 7505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:44.532523 7505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:44.536401 7505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:44.536679 7505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:44.537647 7505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:44.538991 7505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:44.539113 7505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:44.539230 7505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:44.539317 7505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:44.575968 7505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35983
I20260926 21:30:44.576051 7637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35983 every 8 connection(s)
I20260926 21:30:44.577414 7505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:44.581609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 7505
I20260926 21:30:44.581905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 6856
Sep 26 21:30:44 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 26 21:30:44 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458244, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 21:30:44 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458244, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 21:30:44 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458244, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:44.632103 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:44.582969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.0.97.3:36471)
0926 21:30:44.609112 (+ 26143us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:44.609125 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:44.609167 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:30:44.609258 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:44.609280 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:44.609621 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:44.612786 (+ 3165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:44.613086 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.614877 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.616814 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.617711 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.618492 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.623781 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:44.623801 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:44.623819 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:44.623863 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:44.628552 (+ 4689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.629457 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.629467 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.629473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.629679 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.630124 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.630130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.630134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.630581 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:44.630895 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:44.631238 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 21:30:44.631524 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25901,"thread_start_us":26815,"threads_started":1}
I20260926 21:30:44.633445 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:44.580879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38309 (local address 127.0.97.60:40023)
0926 21:30:44.615550 (+ 34671us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:44.615556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:44.615574 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:44.615640 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:44.615645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:44.615719 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:44.615828 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:44.618075 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.619040 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.620362 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.620899 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.621035 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.626727 (+ 5692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:44.626734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:44.626738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:44.626782 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:44.630029 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.630997 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.631002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.631005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.631146 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.631655 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.631660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.631663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.631907 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:44.632069 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:44.632860 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 21:30:44.633068 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34542,"thread_start_us":35046,"threads_started":1}
I20260926 21:30:44.633946 6979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:44.634774 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:44.581106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40023 (local address 127.0.97.2:38309)
0926 21:30:44.614758 (+ 33652us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:44.614777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:44.614894 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:44.615886 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:44.615892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:44.617466 (+ 1574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:44.617887 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:44.617900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.619187 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.619194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.620164 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:44.620174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.621084 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.621089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.621450 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.624028 (+ 2578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:44.624053 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:44.626549 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:44.630181 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.630189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.630209 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.630856 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.631288 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.631292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.631296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.631504 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.632039 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:44.632047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:44.632495 (+ 448us) client_negotiation.cc:769] Sending connection context
0926 21:30:44.632783 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 21:30:44.633285 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33493,"mutex_wait_us":1978,"spinlock_wait_cycles":2432,"thread_start_us":23346,"threads_started":1}
I20260926 21:30:44.635666 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:44.581166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42471 (local address 127.0.97.61:36515)
0926 21:30:44.607715 (+ 26549us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:44.607724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:44.607753 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:30:44.610694 (+ 2941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:44.610702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:44.610815 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:44.610915 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:44.615058 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.616119 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.619797 (+ 3678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.620569 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.620682 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.626168 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:44.626174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:44.626179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:44.626231 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:44.631010 (+ 4779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.631785 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.631791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.631795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.631937 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.632373 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.632377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.632380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.633102 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:44.633225 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:44.634901 (+ 1676us) server_negotiation.cc:299] Negotiation successful
0926 21:30:44.635210 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26402,"thread_start_us":26306,"threads_started":1}
I20260926 21:30:44.636005 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:44.581106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.97.2:42471)
0926 21:30:44.604615 (+ 23509us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:44.604696 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:44.610269 (+ 5573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:44.611581 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:44.611598 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:44.612507 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:44.613444 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:44.613473 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.617165 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.617175 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.619618 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:44.619632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.621069 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.621078 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.621454 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.622303 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:44.622326 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:44.625977 (+ 3651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:44.631194 (+ 5217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.631199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.631203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.631656 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.632064 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.632069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.632074 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.632238 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.633281 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:44.633285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:44.633444 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 21:30:44.633567 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 21:30:44.633869 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23075,"mutex_wait_us":59,"thread_start_us":33791,"threads_started":1}
I20260926 21:30:44.636894 7638 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40023
I20260926 21:30:44.637209 7638 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:44.638072 7638 heartbeater.cc:511] Master 127.0.97.60:40023 requested a full tablet report, sending...
I20260926 21:30:44.639518 7640 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36515
I20260926 21:30:44.639647 7640 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:44.639880 7314 ts_manager.cc:194] Registered new tserver with Master: 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2:35983)
I20260926 21:30:44.640085 7640 heartbeater.cc:511] Master 127.0.97.61:36515 requested a full tablet report, sending...
I20260926 21:30:44.641482 7127 ts_manager.cc:194] Registered new tserver with Master: 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2:35983)
I20260926 21:30:44.646224 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:44.580540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56571 (local address 127.0.97.62:37433)
0926 21:30:44.616352 (+ 35812us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:44.616359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:44.616378 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:30:44.616433 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:44.616440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:44.616522 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:44.616614 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:44.618320 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.623103 (+ 4783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.625977 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.626625 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.626765 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.636305 (+ 9540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:44.636313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:44.636318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:44.636376 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:44.643542 (+ 7166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.644372 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.644376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.644379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.644495 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:44.644912 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:44.644915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:44.644917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:44.645154 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:44.645339 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:44.645690 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 21:30:44.645928 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35676,"thread_start_us":36107,"threads_started":1}
I20260926 21:30:44.646462 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:44.581106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.2:56571)
0926 21:30:44.614058 (+ 32952us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:44.614081 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:44.614208 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:44.616638 (+ 2430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:44.616642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:44.617805 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:44.618135 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:44.618146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.623274 (+ 5128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.623282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.625423 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:44.625437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:44.626804 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:44.626812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:44.626987 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:44.627469 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:44.627484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:44.630034 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:44.643710 (+ 13676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.643716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.643721 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.644251 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.644614 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:44.644620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:44.644624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:44.644810 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:44.645303 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:44.645308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:44.645447 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 21:30:44.645616 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 21:30:44.645949 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32770,"mutex_wait_us":32,"spinlock_wait_cycles":27008,"thread_start_us":33180,"threads_started":1}
I20260926 21:30:44.647276 7639 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37433
I20260926 21:30:44.647423 7639 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:44.647800 7639 heartbeater.cc:511] Master 127.0.97.62:37433 requested a full tablet report, sending...
I20260926 21:30:44.648759 6528 ts_manager.cc:194] Re-registered known tserver with Master: 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2:35983)
I20260926 21:30:44.649631 6528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56571
I20260926 21:30:44.721930 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 6856
I20260926 21:30:44.738966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36471
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37433
--webserver_port=33293
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515,127.0.97.60:40023 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:30:45.025341 7494 heartbeater.cc:503] Master 127.0.97.62:37433 was elected leader, sending a full tablet report...
W20260926 21:30:45.074236 7651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:45.074565 7651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:45.074673 7651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:45.081969 7651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:45.082072 7651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:45.082181 7651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:45.091876 7651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33293
--tserver_master_addrs=127.0.97.62:37433,127.0.97.61:36515,127.0.97.60:40023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:45.093129 7651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:45.094359 7651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:45.107442 7657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:45.107465 7656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:45.107980 7659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:45.108295 7651 server_base.cc:1034] running on GCE node
I20260926 21:30:45.108906 7651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:45.109859 7651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:45.111073 7651 hybrid_clock.cc:648] HybridClock initialized: now 1790458245111026 us; error 63 us; skew 500 ppm
I20260926 21:30:45.111388 7651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:45 dist-test-slave-87bt krb5kdc[6483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:45.114322 7651 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 21:30:45.115748 7651 webserver.cc:460] Webserver started at http://127.0.97.3:33293/ using document root <none> and password file <none>
I20260926 21:30:45.116376 7651 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:45.116483 7651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:45.121842 7651 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 21:30:45.124477 7666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:45.125759 7651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:45.125885 7651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "36515a88910b4fd99cccc5f43c2ba661"
format_stamp: "Formatted at 2026-09-26 21:30:41 on dist-test-slave-87bt"
server_key: "d61d83436f88f80372102e53aa78380d"
server_key_iv: "e94bc946609a254691c42d27ec960b53"
server_key_version: "encryptionkey@0"
I20260926 21:30:45.126844 7651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:45.140329 7651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:45.144361 7651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:45.144588 7651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:45.145453 7651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:45.146934 7651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:45.147053 7651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:45.147168 7651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:45.147258 7651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:45.181277 7651 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36471
I20260926 21:30:45.181335 7778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36471 every 8 connection(s)
I20260926 21:30:45.182832 7651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:45.188951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 7651
Sep 26 21:30:45 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 21:30:45 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 21:30:45 dist-test-slave-87bt krb5kdc[6483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 21:30:45.229384 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.186059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53633 (local address 127.0.97.62:37433)
0926 21:30:45.198940 (+ 12881us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.198951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.200158 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 21:30:45.202841 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.202848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.202938 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.203030 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.206055 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.207210 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.210935 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.211559 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.211695 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.216521 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.216532 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.216540 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.216597 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.221306 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.224160 (+ 2854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.224169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.224174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.226079 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.226731 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.226739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.226745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.227169 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.227298 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.228757 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.229028 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12689,"thread_start_us":12561,"threads_started":1}
I20260926 21:30:45.233891 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.186940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.97.3:45369)
0926 21:30:45.202964 (+ 16024us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:45.202992 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:45.203110 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:45.203605 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:45.203619 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:45.204680 (+ 1061us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:45.205845 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.205872 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.207409 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.207416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.209892 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.209902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.211190 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.211198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.211495 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.212338 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:45.212368 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:45.218377 (+ 6009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:45.226075 (+ 7698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:45.226085 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:45.226092 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:45.226896 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:45.229608 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:45.229617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:45.229624 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:45.229857 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:45.230920 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:45.230930 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:45.231175 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 21:30:45.231361 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 21:30:45.231769 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15889,"mutex_wait_us":137,"thread_start_us":15702,"threads_started":1}
I20260926 21:30:45.235500 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.186982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45369 (local address 127.0.97.61:36515)
0926 21:30:45.200551 (+ 13569us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.200556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.203105 (+ 2549us) server_negotiation.cc:407] Connection header received
0926 21:30:45.203211 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.203227 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.203344 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.203490 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.206056 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.207211 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.210065 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.210758 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.210865 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.218597 (+ 7732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.218607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.218613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.218662 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.225834 (+ 7172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.229120 (+ 3286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.229130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.229137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.229366 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.230090 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.230118 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.230124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.230611 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.230789 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.234768 (+ 3979us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.235025 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13462,"thread_start_us":13354,"threads_started":1}
I20260926 21:30:45.240084 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.187238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40023 (local address 127.0.97.3:47059)
0926 21:30:45.205247 (+ 18009us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:45.205275 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:45.205464 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:45.207454 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:45.207459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:45.208129 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:45.208442 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.208454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.209892 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.209897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.212475 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.212494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.213694 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.213700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.213928 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.214866 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:45.214890 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:45.220555 (+ 5665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:45.236920 (+ 16365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:45.236927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:45.236932 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:45.237500 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:45.238030 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:45.238039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:45.238044 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:45.238220 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:45.238995 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:45.239001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:45.239154 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 21:30:45.239282 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 21:30:45.239591 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17835,"mutex_wait_us":4,"thread_start_us":12052,"threads_started":1}
I20260926 21:30:45.242033 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.186392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47059 (local address 127.0.97.60:40023)
0926 21:30:45.188531 (+ 2139us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.188541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.205374 (+ 16833us) server_negotiation.cc:407] Connection header received
0926 21:30:45.205633 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.205640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.205753 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.205836 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.208625 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.209706 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.212649 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.213293 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.213397 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.220814 (+ 7417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.220824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.220832 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.220894 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.236510 (+ 15616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.237701 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.237707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.237711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.237880 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.238390 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.238395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.238398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.238821 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.241242 (+ 2421us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.241374 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.241612 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1975,"thread_start_us":1823,"threads_started":1}
I20260926 21:30:45.242710 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.186591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.3:53633)
0926 21:30:45.199872 (+ 13281us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:45.199965 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:45.202347 (+ 2382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:45.203605 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:45.203619 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:45.204691 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:45.205847 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.205875 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.208437 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.208445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.210762 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.210782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.211739 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.211747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.211967 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.212549 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:45.212571 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:45.216180 (+ 3609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:45.221601 (+ 5421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:45.221615 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:45.221642 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:45.224030 (+ 2388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:45.226195 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:45.226203 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:45.226210 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:45.226615 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:45.227665 (+ 1050us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:45.227678 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:45.228300 (+ 622us) client_negotiation.cc:769] Sending connection context
0926 21:30:45.228672 (+ 372us) client_negotiation.cc:240] Negotiation successful
0926 21:30:45.229216 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12816,"mutex_wait_us":73,"spinlock_wait_cycles":27392,"thread_start_us":18231,"threads_started":1}
I20260926 21:30:45.244810 7779 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40023
I20260926 21:30:45.245152 7779 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:45.245329 7780 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37433
I20260926 21:30:45.245479 7780 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:45.246286 7779 heartbeater.cc:511] Master 127.0.97.60:40023 requested a full tablet report, sending...
I20260926 21:30:45.247601 7780 heartbeater.cc:511] Master 127.0.97.62:37433 requested a full tablet report, sending...
I20260926 21:30:45.249426 7314 ts_manager.cc:194] Registered new tserver with Master: 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471)
I20260926 21:30:45.250030 6528 ts_manager.cc:194] Re-registered known tserver with Master: 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471)
I20260926 21:30:45.251156 6528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53633
I20260926 21:30:45.251217 7781 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36515
I20260926 21:30:45.251338 7781 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:45.251799 7781 heartbeater.cc:511] Master 127.0.97.61:36515 requested a full tablet report, sending...
I20260926 21:30:45.253067 7127 ts_manager.cc:194] Registered new tserver with Master: 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471)
I20260926 21:30:45.259337 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.219995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.0.97.60:40023)
0926 21:30:45.223453 (+ 3458us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.223457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.235034 (+ 11577us) server_negotiation.cc:407] Connection header received
0926 21:30:45.235207 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.235214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.235301 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.235418 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.238171 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.239367 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.243333 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.244007 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.244125 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.249079 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.249088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.249094 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.249154 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.255098 (+ 5944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.256392 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.256402 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.256407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.256588 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.257070 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.257075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.257079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.257477 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.257734 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.258753 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.258959 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3352,"thread_start_us":3227,"threads_started":1}
I20260926 21:30:45.259330 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.213691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60470 (local address 127.0.97.62:37433)
0926 21:30:45.219055 (+ 5364us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.219061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.225986 (+ 6925us) server_negotiation.cc:407] Connection header received
0926 21:30:45.227221 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.227228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.227323 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.227414 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.236195 (+ 8781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.237688 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.242752 (+ 5064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.243872 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.243962 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.249491 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.249497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.249503 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.249545 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.256247 (+ 6702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.257008 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.257017 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.257021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.257191 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.257704 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.257710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.257713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.258064 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.258267 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.258680 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.258930 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5223,"thread_start_us":6066,"threads_started":1}
I20260926 21:30:45.264134 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.224537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.97.61:36515)
0926 21:30:45.235769 (+ 11232us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.235776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.243456 (+ 7680us) server_negotiation.cc:407] Connection header received
0926 21:30:45.243733 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.243739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.243824 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.243899 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.249794 (+ 5895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.251501 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.253773 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.254662 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.254846 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.258195 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.258201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.258206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.258243 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.261551 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.262242 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.262248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.262251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.262382 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.262784 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.262789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.262792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.263052 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.263297 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.263599 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.263821 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11112,"thread_start_us":25649,"threads_started":1}
I20260926 21:30:45.263716 6528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:45.271529 6528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:45.321776 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.292009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36471 (local address 127.0.0.1:32962)
0926 21:30:45.301812 (+ 9803us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:45.301840 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:45.302008 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:45.314467 (+ 12459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:45.314475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:45.314507 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:45.315231 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.315248 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.317129 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.317134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.318776 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.318790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.320270 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.320276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.320496 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.320544 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 21:30:45.320637 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 21:30:45.320784 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9615,"mutex_wait_us":3,"thread_start_us":26478,"threads_started":1}
I20260926 21:30:45.322073 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.292617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.0.97.3:36471)
0926 21:30:45.313883 (+ 21266us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.313890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.313919 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:30:45.314001 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.314007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.314217 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.314681 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:45.315407 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.317002 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.319010 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.320096 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.320239 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.320471 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.321191 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.321384 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21105,"thread_start_us":21823,"threads_started":1}
I20260926 21:30:45.326704 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.310022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35983 (local address 127.0.0.1:36292)
0926 21:30:45.315777 (+ 5755us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:45.315801 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:45.315912 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:45.316971 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:45.316979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:45.317021 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:45.317704 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.317731 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.321102 (+ 3371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.321108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.323672 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.323687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.325883 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.325891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.326064 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.326100 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 21:30:45.326169 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 21:30:45.326302 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5633,"mutex_wait_us":25,"thread_start_us":592,"threads_started":1}
I20260926 21:30:45.327123 7714 tablet_service.cc:1467] Processing CreateTablet for tablet 0297786aad194aa190475360edb8f0ab (DEFAULT_TABLE table=test-table [id=640358d9cfa24109ac139100b5d296b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:45.327744 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.303104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.97.2:35983)
0926 21:30:45.316343 (+ 13239us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.316352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.316380 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:30:45.316473 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.316480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.316732 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.316926 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:45.318998 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.320939 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.323889 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.325240 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.327050 (+ 1810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.327273 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.327340 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.327488 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13070,"thread_start_us":12983,"threads_started":1}
I20260926 21:30:45.327773 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.292741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35291 (local address 127.0.0.1:54062)
0926 21:30:45.310763 (+ 18022us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:45.310788 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:45.310959 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:45.321781 (+ 10822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:45.321785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:45.321817 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:45.322166 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.322181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.323603 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.323612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.324942 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.324953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.326933 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.326938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.327150 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.327190 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 21:30:45.327281 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 21:30:45.327404 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17831,"mutex_wait_us":25,"thread_start_us":9564,"threads_started":1}
I20260926 21:30:45.328500 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.293004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.0.97.1:35291)
0926 21:30:45.320901 (+ 27897us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.320912 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.320941 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:30:45.321038 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.321048 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.321319 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.321527 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:45.322403 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.323473 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.325097 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.326479 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.326593 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.326788 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.328077 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.328228 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27722,"thread_start_us":27615,"threads_started":1}
I20260926 21:30:45.328642 7714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0297786aad194aa190475360edb8f0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:45.331931 7573 tablet_service.cc:1467] Processing CreateTablet for tablet 0297786aad194aa190475360edb8f0ab (DEFAULT_TABLE table=test-table [id=640358d9cfa24109ac139100b5d296b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:45.332093 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 0297786aad194aa190475360edb8f0ab (DEFAULT_TABLE table=test-table [id=640358d9cfa24109ac139100b5d296b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:45.333184 7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0297786aad194aa190475360edb8f0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:45.333349 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0297786aad194aa190475360edb8f0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:45.350838 7816 tablet_bootstrap.cc:492] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Bootstrap starting.
I20260926 21:30:45.351926 7818 tablet_bootstrap.cc:492] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Bootstrap starting.
I20260926 21:30:45.355962 7816 tablet_bootstrap.cc:654] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:45.356113 7817 tablet_bootstrap.cc:492] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: Bootstrap starting.
I20260926 21:30:45.356418 7818 tablet_bootstrap.cc:654] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:45.357640 7816 log.cc:824] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:45.359299 7818 log.cc:824] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:45.360508 7817 tablet_bootstrap.cc:654] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:45.361999 7817 log.cc:824] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:45.364850 7816 tablet_bootstrap.cc:492] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: No bootstrap required, opened a new log
I20260926 21:30:45.365079 7818 tablet_bootstrap.cc:492] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: No bootstrap required, opened a new log
I20260926 21:30:45.365099 7817 tablet_bootstrap.cc:492] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: No bootstrap required, opened a new log
I20260926 21:30:45.365131 7816 ts_tablet_manager.cc:1397] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260926 21:30:45.365381 7817 ts_tablet_manager.cc:1397] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 21:30:45.365378 7818 ts_tablet_manager.cc:1397] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260926 21:30:45.373409 7816 raft_consensus.cc:348] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.373786 7816 raft_consensus.cc:374] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:45.373878 7816 raft_consensus.cc:729] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36515a88910b4fd99cccc5f43c2ba661, State: Initialized, Role: FOLLOWER
I20260926 21:30:45.375002 7816 consensus_queue.cc:260] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.376138 7817 raft_consensus.cc:348] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.376571 7817 raft_consensus.cc:374] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:45.376667 7817 raft_consensus.cc:729] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25878a5b4bad407f9592fdb2b6ce6d46, State: Initialized, Role: FOLLOWER
I20260926 21:30:45.376453 7818 raft_consensus.cc:348] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.376869 7818 raft_consensus.cc:374] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:45.376961 7818 raft_consensus.cc:729] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47732b6aa48c462cb58701cc6c40eb7d, State: Initialized, Role: FOLLOWER
I20260926 21:30:45.377280 7817 consensus_queue.cc:260] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.377471 7818 consensus_queue.cc:260] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.379472 7817 ts_tablet_manager.cc:1428] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 21:30:45.379833 7639 heartbeater.cc:503] Master 127.0.97.62:37433 was elected leader, sending a full tablet report...
I20260926 21:30:45.383076 7816 ts_tablet_manager.cc:1428] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260926 21:30:45.385686 7780 heartbeater.cc:503] Master 127.0.97.62:37433 was elected leader, sending a full tablet report...
I20260926 21:30:45.387701 7818 ts_tablet_manager.cc:1428] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Time spent starting tablet: real 0.022s user 0.007s sys 0.015s
W20260926 21:30:45.441737 7782 tablet.cc:2369] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:45.472481 7824 raft_consensus.cc:482] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:45.472847 7824 raft_consensus.cc:504] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.475054 7824 leader_election.cc:288] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471), 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2:35983)
I20260926 21:30:45.483486 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.475368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43483 (local address 127.0.97.3:36471)
0926 21:30:45.475610 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.475620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.477132 (+ 1512us) server_negotiation.cc:407] Connection header received
0926 21:30:45.477283 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.477286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.477343 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.477446 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:45.477898 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.478840 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.481133 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.482334 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.482464 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.482550 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.483116 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.483252 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:30:45.483486 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.475623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46341 (local address 127.0.97.2:35983)
0926 21:30:45.475847 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.475854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.476434 (+ 580us) server_negotiation.cc:407] Connection header received
0926 21:30:45.476650 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.476660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.476750 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.476851 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:45.477456 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.478491 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.481006 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.482255 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.482399 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.482521 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.483116 (+ 595us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.483241 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:30:45.483732 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.475315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35983 (local address 127.0.97.1:46341)
0926 21:30:45.476324 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:45.476339 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:45.476494 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:45.476893 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:45.476899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:45.476936 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:45.477323 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.477332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.478619 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.478623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.480838 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.480853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.482530 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.482537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.482837 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.482985 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 21:30:45.483080 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 21:30:45.483274 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":165,"thread_start_us":717,"threads_started":1}
I20260926 21:30:45.484087 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.475444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36471 (local address 127.0.97.1:43483)
0926 21:30:45.477007 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:45.477026 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:45.477150 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:45.477472 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:45.477476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:45.477506 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:45.477744 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.477750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.478951 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.478956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.480999 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:45.481009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.482541 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.482546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:45.482837 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.482984 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 21:30:45.483080 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 21:30:45.483274 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1411,"mutex_wait_us":158,"thread_start_us":1273,"threads_started":1}
I20260926 21:30:45.484957 7734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0297786aad194aa190475360edb8f0ab" candidate_uuid: "47732b6aa48c462cb58701cc6c40eb7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36515a88910b4fd99cccc5f43c2ba661" is_pre_election: true
W20260926 21:30:45.485191 7496 tablet.cc:2369] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:45.484745 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0297786aad194aa190475360edb8f0ab" candidate_uuid: "47732b6aa48c462cb58701cc6c40eb7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" is_pre_election: true
I20260926 21:30:45.485584 7593 raft_consensus.cc:2454] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47732b6aa48c462cb58701cc6c40eb7d in term 0.
I20260926 21:30:45.485550 7734 raft_consensus.cc:2454] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47732b6aa48c462cb58701cc6c40eb7d in term 0.
I20260926 21:30:45.486251 7381 leader_election.cc:302] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25878a5b4bad407f9592fdb2b6ce6d46, 36515a88910b4fd99cccc5f43c2ba661, 47732b6aa48c462cb58701cc6c40eb7d; no voters:
I20260926 21:30:45.486680 7824 raft_consensus.cc:2781] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:45.486820 7824 raft_consensus.cc:482] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:45.486946 7824 raft_consensus.cc:3037] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:45.489068 7824 raft_consensus.cc:504] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.489689 7824 leader_election.cc:288] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [CANDIDATE]: Term 1 election: Requested vote from peers 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471), 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2:35983)
I20260926 21:30:45.490165 7734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0297786aad194aa190475360edb8f0ab" candidate_uuid: "47732b6aa48c462cb58701cc6c40eb7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36515a88910b4fd99cccc5f43c2ba661"
I20260926 21:30:45.490208 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0297786aad194aa190475360edb8f0ab" candidate_uuid: "47732b6aa48c462cb58701cc6c40eb7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25878a5b4bad407f9592fdb2b6ce6d46"
I20260926 21:30:45.490389 7593 raft_consensus.cc:3037] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:45.490386 7734 raft_consensus.cc:3037] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:45.493618 7593 raft_consensus.cc:2454] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47732b6aa48c462cb58701cc6c40eb7d in term 1.
I20260926 21:30:45.493618 7734 raft_consensus.cc:2454] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47732b6aa48c462cb58701cc6c40eb7d in term 1.
I20260926 21:30:45.494016 7381 leader_election.cc:302] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25878a5b4bad407f9592fdb2b6ce6d46, 36515a88910b4fd99cccc5f43c2ba661, 47732b6aa48c462cb58701cc6c40eb7d; no voters:
I20260926 21:30:45.494307 7824 raft_consensus.cc:2781] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:45.494778 7824 raft_consensus.cc:686] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 1 LEADER]: Becoming Leader. State: Replica: 47732b6aa48c462cb58701cc6c40eb7d, State: Running, Role: LEADER
I20260926 21:30:45.495395 7824 consensus_queue.cc:237] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } }
I20260926 21:30:45.500365 6528 catalog_manager.cc:5495] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "47732b6aa48c462cb58701cc6c40eb7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } health_report { overall_health: UNKNOWN } } }
I20260926 21:30:45.521994 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.517251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.0.97.1:35291)
0926 21:30:45.517407 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.517412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.517427 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:30:45.517545 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.517549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.517617 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.517737 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:45.517998 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.518749 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.519986 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.520399 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.520501 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.520945 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:45.521511 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:45.521616 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.521725 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.521796 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 21:30:45.530283 7593 raft_consensus.cc:1261] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 FOLLOWER]: Refusing update from remote peer 47732b6aa48c462cb58701cc6c40eb7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:45.530282 7734 raft_consensus.cc:1261] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 1 FOLLOWER]: Refusing update from remote peer 47732b6aa48c462cb58701cc6c40eb7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:45.531296 7827 consensus_queue.cc:1035] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:45.531642 7824 consensus_queue.cc:1035] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:45.548518 7831 mvcc.cc:205] Tried to move back new op lower bound from 7333716973682544640 to 7333716973552484352. Current Snapshot: MvccSnapshot[applied={T|T < 7333716973682544640}]
I20260926 21:30:45.579926 6528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:45.580181 6528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:45.589268 6528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05fc3954f5e743849b63c1bb2e6228ae: Sending DeleteTablet for 3 replicas of tablet 0297786aad194aa190475360edb8f0ab
I20260926 21:30:45.590395 7714 tablet_service.cc:1514] Processing DeleteTablet for tablet 0297786aad194aa190475360edb8f0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260926 21:30:45.590459 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet 0297786aad194aa190475360edb8f0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54062
I20260926 21:30:45.593048 7573 tablet_service.cc:1514] Processing DeleteTablet for tablet 0297786aad194aa190475360edb8f0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36292
I20260926 21:30:45.604628 7843 tablet_replica.cc:330] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: stopping tablet replica
I20260926 21:30:45.606474 7843 raft_consensus.cc:2229] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:45.607937 7843 raft_consensus.cc:2258] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:45.612140 7843 ts_tablet_manager.cc:1905] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:45.615331 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 6496
I20260926 21:30:45.615430 7842 tablet_replica.cc:330] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: stopping tablet replica
I20260926 21:30:45.616438 6558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:45.616658 7841 tablet_replica.cc:330] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: stopping tablet replica
I20260926 21:30:45.617161 7841 raft_consensus.cc:2229] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:45.617786 7841 raft_consensus.cc:2258] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:45.618757 7842 raft_consensus.cc:2229] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:45.619509 7842 raft_consensus.cc:2258] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:45.622278 7842 ts_tablet_manager.cc:1905] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:45.622732 7841 ts_tablet_manager.cc:1905] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:45.635156 7843 ts_tablet_manager.cc:1918] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:45.635335 7843 log.cc:1196] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/0297786aad194aa190475360edb8f0ab
I20260926 21:30:45.635401 7841 ts_tablet_manager.cc:1918] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:45.635561 7841 log.cc:1196] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/0297786aad194aa190475360edb8f0ab
I20260926 21:30:45.635907 7843 ts_tablet_manager.cc:1939] T 0297786aad194aa190475360edb8f0ab P 25878a5b4bad407f9592fdb2b6ce6d46: Deleting consensus metadata
I20260926 21:30:45.636384 7841 ts_tablet_manager.cc:1939] T 0297786aad194aa190475360edb8f0ab P 36515a88910b4fd99cccc5f43c2ba661: Deleting consensus metadata
I20260926 21:30:45.639276 7842 ts_tablet_manager.cc:1918] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:45.639477 7842 log.cc:1196] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/0297786aad194aa190475360edb8f0ab
I20260926 21:30:45.640067 7842 ts_tablet_manager.cc:1939] T 0297786aad194aa190475360edb8f0ab P 47732b6aa48c462cb58701cc6c40eb7d: Deleting consensus metadata
I20260926 21:30:45.640769 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.595728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.0.97.61:36515)
0926 21:30:45.595944 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.595953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.613674 (+ 17721us) server_negotiation.cc:407] Connection header received
0926 21:30:45.613839 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.613845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.613927 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.614039 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.621669 (+ 7630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.623474 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.626055 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.626710 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.626846 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.634000 (+ 7154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.634007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.634013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.634057 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.637504 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.638297 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.638303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.638307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.638484 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.638938 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.638943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.638947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.639313 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.639561 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.640117 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.640353 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:30:45.655366 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.614302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.0.97.60:40023)
0926 21:30:45.614511 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.614517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.642271 (+ 27754us) server_negotiation.cc:407] Connection header received
0926 21:30:45.642560 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.642565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.642629 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.642748 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.644427 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.645126 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.646767 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.647340 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.647492 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.650135 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.650146 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.650149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.650187 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.652771 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.653387 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.653394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.653397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.653523 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.653891 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.653897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.653900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.654157 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.654409 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.654749 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.654967 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 21:30:45.821401 6512 catalog_manager.cc:4841] TS 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1:35291): tablet 0297786aad194aa190475360edb8f0ab (table test-table [id=640358d9cfa24109ac139100b5d296b3]) successfully deleted
I20260926 21:30:45.822227 6513 catalog_manager.cc:4841] TS 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471): tablet 0297786aad194aa190475360edb8f0ab (table test-table [id=640358d9cfa24109ac139100b5d296b3]) successfully deleted
I20260926 21:30:45.822525 6512 catalog_manager.cc:4841] TS 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2:35983): tablet 0297786aad194aa190475360edb8f0ab (table test-table [id=640358d9cfa24109ac139100b5d296b3]) successfully deleted
I20260926 21:30:45.822790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 6496
I20260926 21:30:45.886152 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.849896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.0.97.61:36515)
0926 21:30:45.850158 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.850166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.866877 (+ 16711us) server_negotiation.cc:407] Connection header received
0926 21:30:45.867700 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.867709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.867835 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.867928 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.871688 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.872908 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.875328 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.875841 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.875992 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.879583 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.879588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.879591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.879631 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.883115 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.883942 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.883948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.883951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.884098 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.884490 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.884496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.884499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.884886 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.885139 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.885566 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.885784 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:30:45.887113 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:45.849519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.0.97.60:40023)
0926 21:30:45.849727 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:45.849733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:45.870986 (+ 21253us) server_negotiation.cc:407] Connection header received
0926 21:30:45.871152 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:45.871157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:45.871246 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:45.871353 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:45.873784 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.874865 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.876699 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:45.877173 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:45.877270 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:45.880671 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:45.880680 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:45.880686 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:45.880733 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:45.883983 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.884765 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.884771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.884774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.884905 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:45.885348 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:45.885355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:45.885358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:45.885738 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:45.885972 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:45.886517 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 21:30:45.886757 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:30:46.563180 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:46.562481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.3:54553)
0926 21:30:46.562882 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:46.562992 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":208,"threads_started":1}
W20260926 21:30:46.563548 7780 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37433 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
I20260926 21:30:46.569936 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:46.569194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.1:41137)
0926 21:30:46.569632 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:46.569774 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":246,"threads_started":1}
W20260926 21:30:46.570350 7494 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37433 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
I20260926 21:30:46.571240 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:46.570346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.2:59707)
0926 21:30:46.570838 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:46.570948 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":280,"threads_started":1}
W20260926 21:30:46.571621 7639 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37433 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
I20260926 21:30:47.088115 7856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05fc3954f5e743849b63c1bb2e6228ae)
I20260926 21:30:47.088351 7856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } }
I20260926 21:30:47.088901 7857 raft_consensus.cc:482] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05fc3954f5e743849b63c1bb2e6228ae)
I20260926 21:30:47.089188 7857 raft_consensus.cc:504] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } }
I20260926 21:30:47.089392 7856 leader_election.cc:288] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05fc3954f5e743849b63c1bb2e6228ae (127.0.97.62:37433), b638d68f18a54b75b5c9c83ed9be836e (127.0.97.61:36515)
I20260926 21:30:47.099402 7857 leader_election.cc:288] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05fc3954f5e743849b63c1bb2e6228ae (127.0.97.62:37433), 3840b9c4dd1a40f1ba8be8abc9cfcbcd (127.0.97.60:40023)
I20260926 21:30:47.099803 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.091174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54130)
0926 21:30:47.099196 (+ 8022us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.099457 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7838,"threads_started":1}
W20260926 21:30:47.102689 7113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
I20260926 21:30:47.103003 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.089488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54122)
0926 21:30:47.102463 (+ 12975us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.102651 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12828,"thread_start_us":12885,"threads_started":1}
W20260926 21:30:47.104312 7300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
I20260926 21:30:47.104871 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.104203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54134)
0926 21:30:47.104392 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.104596 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 21:30:47.105154 7113 leader_election.cc:334] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05fc3954f5e743849b63c1bb2e6228ae (127.0.97.62:37433): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
I20260926 21:30:47.106294 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.105871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54140)
0926 21:30:47.106046 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.106145 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 21:30:47.106527 7300 leader_election.cc:334] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05fc3954f5e743849b63c1bb2e6228ae (127.0.97.62:37433): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
I20260926 21:30:47.110234 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.089625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36515 (local address 127.0.0.1:55058)
0926 21:30:47.102754 (+ 13129us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.102774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:47.102938 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:47.103454 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:47.103463 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:47.103496 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:47.103986 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.104006 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.105799 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.105810 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.107836 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.107851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.109348 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.109355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.109605 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.109638 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:30:47.109725 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 21:30:47.109844 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12996,"mutex_wait_us":1,"thread_start_us":12196,"threads_started":1}
I20260926 21:30:47.110809 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.089765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.0.97.61:36515)
0926 21:30:47.102352 (+ 12587us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.102360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.103098 (+ 738us) server_negotiation.cc:407] Connection header received
0926 21:30:47.103183 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.103191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.103281 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.103736 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:47.104238 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.105644 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.107981 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.108997 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.109121 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.109233 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.110084 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.110212 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":12418,"thread_start_us":12214,"threads_started":1}
I20260926 21:30:47.111304 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b638d68f18a54b75b5c9c83ed9be836e" is_pre_election: true
I20260926 21:30:47.111621 7136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3840b9c4dd1a40f1ba8be8abc9cfcbcd in term 1.
I20260926 21:30:47.112073 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.096921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40023 (local address 127.0.0.1:46798)
0926 21:30:47.100045 (+ 3124us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.100066 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:47.100243 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:47.105438 (+ 5195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:47.105443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:47.105468 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:47.105846 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.105859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.107349 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.107354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.108938 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.108956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.111330 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.111334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.111446 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.111507 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 21:30:47.111591 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 21:30:47.111736 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3002,"thread_start_us":1865,"threads_started":1}
I20260926 21:30:47.112116 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.102662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.0.97.60:40023)
0926 21:30:47.105109 (+ 2447us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.105115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.105136 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:30:47.105199 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.105204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.105289 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.105389 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:47.106062 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.107219 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.109124 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.111147 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.111285 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.111392 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.111625 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.111747 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":965,"server-negotiator.queue_time_us":1528,"thread_start_us":1417,"threads_started":1}
I20260926 21:30:47.112146 7301 leader_election.cc:302] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3840b9c4dd1a40f1ba8be8abc9cfcbcd, b638d68f18a54b75b5c9c83ed9be836e; no voters: 05fc3954f5e743849b63c1bb2e6228ae
I20260926 21:30:47.112600 7856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 21:30:47.112731 7856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 05fc3954f5e743849b63c1bb2e6228ae)
I20260926 21:30:47.112846 7856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:30:47.112818 7324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b638d68f18a54b75b5c9c83ed9be836e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" is_pre_election: true
I20260926 21:30:47.115010 7856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } }
I20260926 21:30:47.115424 7324 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b638d68f18a54b75b5c9c83ed9be836e in current term 2: Already voted for candidate 3840b9c4dd1a40f1ba8be8abc9cfcbcd in this term.
I20260926 21:30:47.115682 7856 leader_election.cc:288] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [CANDIDATE]: Term 2 election: Requested vote from peers 05fc3954f5e743849b63c1bb2e6228ae (127.0.97.62:37433), b638d68f18a54b75b5c9c83ed9be836e (127.0.97.61:36515)
I20260926 21:30:47.116204 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.115822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54144)
0926 21:30:47.115970 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.116067 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 21:30:47.116129 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b638d68f18a54b75b5c9c83ed9be836e"
I20260926 21:30:47.116343 7136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:30:47.116420 7111 leader_election.cc:302] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b638d68f18a54b75b5c9c83ed9be836e; no voters: 05fc3954f5e743849b63c1bb2e6228ae, 3840b9c4dd1a40f1ba8be8abc9cfcbcd
I20260926 21:30:47.117290 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.116965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54148)
0926 21:30:47.117110 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.117191 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:30:47.117444 7300 leader_election.cc:334] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 05fc3954f5e743849b63c1bb2e6228ae (127.0.97.62:37433): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
I20260926 21:30:47.118707 7136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3840b9c4dd1a40f1ba8be8abc9cfcbcd in term 2.
I20260926 21:30:47.118916 7857 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 21:30:47.119102 7301 leader_election.cc:302] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3840b9c4dd1a40f1ba8be8abc9cfcbcd, b638d68f18a54b75b5c9c83ed9be836e; no voters: 05fc3954f5e743849b63c1bb2e6228ae
I20260926 21:30:47.119398 7856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:30:47.119803 7856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [term 2 LEADER]: Becoming Leader. State: Replica: 3840b9c4dd1a40f1ba8be8abc9cfcbcd, State: Running, Role: LEADER
I20260926 21:30:47.120185 7856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } }
I20260926 21:30:47.121922 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3840b9c4dd1a40f1ba8be8abc9cfcbcd. Latest consensus state: current_term: 2 leader_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:47.122205 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:47.122821 7868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:47.126080 7868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:47.126946 7868 catalog_manager.cc:1259] Loaded cluster ID: b25f3348d34647c6b6f738f663807fdb
I20260926 21:30:47.127044 7868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:47.128150 7868 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:47.128902 7868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Loaded TSK: 0
I20260926 21:30:47.129712 7868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:30:47.165772 7314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:47.172667 7314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:47.176793 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.176378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54172)
0926 21:30:47.176541 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.176668 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 21:30:47.176949 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [term 2 FOLLOWER]: Refusing update from remote peer 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 21:30:47.177774 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.177397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54182)
0926 21:30:47.177581 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.177648 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 21:30:47.177719 7866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 21:30:47.177989 7300 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd -> Peer 05fc3954f5e743849b63c1bb2e6228ae (127.0.97.62:37433): Couldn't send request to peer 05fc3954f5e743849b63c1bb2e6228ae. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 21:30:47.180395 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3840b9c4dd1a40f1ba8be8abc9cfcbcd. Latest consensus state: current_term: 2 leader_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:47.180702 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:30:47.181736 7869 mvcc.cc:205] Tried to move back new op lower bound from 7333716980430127104 to 7333716980207742976. Current Snapshot: MvccSnapshot[applied={T|T < 7333716980430127104}]
I20260926 21:30:47.182209 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:47.182524 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:47.183306 7870 mvcc.cc:205] Tried to move back new op lower bound from 7333716980430127104 to 7333716980207742976. Current Snapshot: MvccSnapshot[applied={T|T < 7333716980430127104}]
I20260926 21:30:47.184494 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "05fc3954f5e743849b63c1bb2e6228ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37433 } } peers { permanent_uuid: "b638d68f18a54b75b5c9c83ed9be836e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36515 } attrs { promote: false } } peers { permanent_uuid: "3840b9c4dd1a40f1ba8be8abc9cfcbcd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40023 } attrs { promote: false } } }
I20260926 21:30:47.184747 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P b638d68f18a54b75b5c9c83ed9be836e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:30:47.200165 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.190836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.0.97.2:35983)
0926 21:30:47.191292 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.191300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.191318 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:47.191386 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.191390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.191462 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.191561 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:47.192487 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.193945 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.197142 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.198895 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.199078 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.199215 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.199724 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.199864 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260926 21:30:47.200151 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.190746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35983 (local address 127.0.0.1:43084)
0926 21:30:47.190901 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.190922 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:47.191040 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:47.191750 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:47.191755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:47.191796 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:47.192282 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.192295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.194175 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.194184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.196861 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.196883 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.199320 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.199330 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.199538 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.199573 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:30:47.199697 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 21:30:47.199840 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"spinlock_wait_cycles":10880}
I20260926 21:30:47.201006 7573 tablet_service.cc:1467] Processing CreateTablet for tablet a45e2702fd2348bc9f3b9aba9931cd33 (DEFAULT_TABLE table=test-table [id=d8d3fdce7a7a4402b8a1f59c0a03bd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:47.201483 7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45e2702fd2348bc9f3b9aba9931cd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:47.207921 7879 tablet_bootstrap.cc:492] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: Bootstrap starting.
I20260926 21:30:47.208074 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.191389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36471 (local address 127.0.0.1:50232)
0926 21:30:47.191647 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.191682 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:47.191813 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:47.200078 (+ 8265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:47.200086 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:47.200144 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:47.200475 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.200485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.202850 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.202856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.204833 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.204849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.207212 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.207219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.207436 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.207493 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 21:30:47.207614 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 21:30:47.207770 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"spinlock_wait_cycles":7808}
I20260926 21:30:47.208597 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.191532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.0.97.3:36471)
0926 21:30:47.199642 (+ 8110us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.199652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.199676 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:30:47.199762 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.199770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.199881 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.200620 (+ 739us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:47.200793 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.202710 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.205012 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.206883 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.207035 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.207283 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.208273 (+ 990us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.208390 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7951,"thread_start_us":7821,"threads_started":1}
I20260926 21:30:47.209300 7714 tablet_service.cc:1467] Processing CreateTablet for tablet a45e2702fd2348bc9f3b9aba9931cd33 (DEFAULT_TABLE table=test-table [id=d8d3fdce7a7a4402b8a1f59c0a03bd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:47.209717 7714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45e2702fd2348bc9f3b9aba9931cd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:47.209722 7879 tablet_bootstrap.cc:654] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:47.211912 7879 tablet_bootstrap.cc:492] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: No bootstrap required, opened a new log
I20260926 21:30:47.212059 7879 ts_tablet_manager.cc:1397] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 21:30:47.212860 7879 raft_consensus.cc:348] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.213153 7879 raft_consensus.cc:374] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:47.213253 7879 raft_consensus.cc:729] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25878a5b4bad407f9592fdb2b6ce6d46, State: Initialized, Role: FOLLOWER
I20260926 21:30:47.213502 7879 consensus_queue.cc:260] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.214022 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.191641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35291 (local address 127.0.0.1:57236)
0926 21:30:47.199134 (+ 7493us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.199155 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:47.199278 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:47.206630 (+ 7352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:47.206636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:47.206664 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:47.207104 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.207216 (+ 112us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.209537 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.209545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.211495 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.211512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.213145 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.213154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.213351 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.213393 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 21:30:47.213466 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 21:30:47.213614 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7372,"thread_start_us":6474,"threads_started":1}
I20260926 21:30:47.214375 7879 ts_tablet_manager.cc:1428] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 21:30:47.214915 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.192099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.0.97.1:35291)
0926 21:30:47.206187 (+ 14088us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.206193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.206212 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:30:47.206283 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.206288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.206360 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.207395 (+ 1035us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:47.207692 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.209374 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.211724 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.212904 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.214279 (+ 1375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.214387 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.214470 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.214616 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13951,"thread_start_us":14359,"threads_started":1}
I20260926 21:30:47.215965 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.214976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.2:49053)
0926 21:30:47.215608 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.215745 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"threads_started":1}
I20260926 21:30:47.216672 7428 tablet_service.cc:1467] Processing CreateTablet for tablet a45e2702fd2348bc9f3b9aba9931cd33 (DEFAULT_TABLE table=test-table [id=d8d3fdce7a7a4402b8a1f59c0a03bd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:47.217159 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45e2702fd2348bc9f3b9aba9931cd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:47.218043 7638 heartbeater.cc:503] Master 127.0.97.60:40023 was elected leader, sending a full tablet report...
I20260926 21:30:47.218277 7883 tablet_bootstrap.cc:492] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: Bootstrap starting.
I20260926 21:30:47.221225 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.220847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.2:46551)
0926 21:30:47.221028 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.221116 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:30:47.221354 7883 tablet_bootstrap.cc:654] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:47.223388 7883 tablet_bootstrap.cc:492] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: No bootstrap required, opened a new log
I20260926 21:30:47.223521 7883 ts_tablet_manager.cc:1397] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 21:30:47.224359 7883 raft_consensus.cc:348] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.224572 7885 tablet_bootstrap.cc:492] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: Bootstrap starting.
I20260926 21:30:47.224597 7883 raft_consensus.cc:374] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:47.224716 7883 raft_consensus.cc:729] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36515a88910b4fd99cccc5f43c2ba661, State: Initialized, Role: FOLLOWER
I20260926 21:30:47.224979 7883 consensus_queue.cc:260] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.225754 7883 ts_tablet_manager.cc:1428] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 21:30:47.225991 7885 tablet_bootstrap.cc:654] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:47.228519 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.225969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.3:49415)
0926 21:30:47.228168 (+ 2199us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.228283 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2044,"thread_start_us":1930,"threads_started":1}
I20260926 21:30:47.228958 7779 heartbeater.cc:503] Master 127.0.97.60:40023 was elected leader, sending a full tablet report...
I20260926 21:30:47.228976 7885 tablet_bootstrap.cc:492] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: No bootstrap required, opened a new log
I20260926 21:30:47.229188 7885 ts_tablet_manager.cc:1397] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 21:30:47.230060 7885 raft_consensus.cc:348] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.230351 7885 raft_consensus.cc:374] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:47.230566 7885 raft_consensus.cc:729] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47732b6aa48c462cb58701cc6c40eb7d, State: Initialized, Role: FOLLOWER
I20260926 21:30:47.230856 7885 consensus_queue.cc:260] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.231705 7885 ts_tablet_manager.cc:1428] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 21:30:47.232988 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.229166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.3:52553)
0926 21:30:47.232730 (+ 3564us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.232817 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3466,"threads_started":1}
I20260926 21:30:47.233820 7493 heartbeater.cc:503] Master 127.0.97.60:40023 was elected leader, sending a full tablet report...
I20260926 21:30:47.235690 7881 raft_consensus.cc:482] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:47.235934 7881 raft_consensus.cc:504] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.238289 7881 leader_election.cc:288] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471), 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1:35291)
I20260926 21:30:47.244858 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.231952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.1:56079)
0926 21:30:47.244428 (+ 12476us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.244564 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12348,"threads_started":1}
I20260926 21:30:47.245108 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.238646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39587 (local address 127.0.97.1:35291)
0926 21:30:47.239131 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.239139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.239157 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:47.239215 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.239220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.239290 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.239378 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:47.240020 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.240926 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.243247 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.244111 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.244286 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.244387 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.244689 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.244819 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
I20260926 21:30:47.245172 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.238636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35291 (local address 127.0.97.2:39587)
0926 21:30:47.238833 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.238848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:47.238943 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:47.239533 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:47.239538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:47.239588 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:47.239854 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.239866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.241079 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.241084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.243045 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.243058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.244320 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.244326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.244524 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.244569 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 21:30:47.244667 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 21:30:47.244802 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":5}
I20260926 21:30:47.245992 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.245607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.1:35187)
0926 21:30:47.245776 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.245840 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 21:30:47.246536 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45e2702fd2348bc9f3b9aba9931cd33" candidate_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47732b6aa48c462cb58701cc6c40eb7d" is_pre_election: true
I20260926 21:30:47.246939 7448 raft_consensus.cc:2454] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25878a5b4bad407f9592fdb2b6ce6d46 in term 0.
I20260926 21:30:47.247403 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.239182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41219 (local address 127.0.97.3:36471)
0926 21:30:47.239357 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.239366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.239386 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:30:47.239626 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.239635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.239732 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.239807 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:47.240666 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.241684 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.244762 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.246336 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.246486 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.246610 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.247027 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.247163 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:30:47.247611 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.238778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36471 (local address 127.0.97.2:41219)
0926 21:30:47.239326 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.239342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:47.239447 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:47.240035 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:47.240042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:47.240070 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:47.240464 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.240478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.241825 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.241833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.244582 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:47.244596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.246620 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.246627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:47.246854 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.246889 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:30:47.246973 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 21:30:47.247144 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":54,"thread_start_us":5627,"threads_started":1}
I20260926 21:30:47.247593 7526 leader_election.cc:302] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25878a5b4bad407f9592fdb2b6ce6d46, 47732b6aa48c462cb58701cc6c40eb7d; no voters:
I20260926 21:30:47.248011 7881 raft_consensus.cc:2781] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:47.248147 7881 raft_consensus.cc:482] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:47.248260 7881 raft_consensus.cc:3037] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:47.248236 7734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45e2702fd2348bc9f3b9aba9931cd33" candidate_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36515a88910b4fd99cccc5f43c2ba661" is_pre_election: true
I20260926 21:30:47.248478 7734 raft_consensus.cc:2454] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25878a5b4bad407f9592fdb2b6ce6d46 in term 0.
I20260926 21:30:47.250569 7881 raft_consensus.cc:504] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.251221 7881 leader_election.cc:288] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [CANDIDATE]: Term 1 election: Requested vote from peers 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471), 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1:35291)
I20260926 21:30:47.251646 7734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45e2702fd2348bc9f3b9aba9931cd33" candidate_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36515a88910b4fd99cccc5f43c2ba661"
I20260926 21:30:47.251660 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45e2702fd2348bc9f3b9aba9931cd33" candidate_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47732b6aa48c462cb58701cc6c40eb7d"
I20260926 21:30:47.251820 7734 raft_consensus.cc:3037] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:47.251833 7448 raft_consensus.cc:3037] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:47.254153 7734 raft_consensus.cc:2454] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25878a5b4bad407f9592fdb2b6ce6d46 in term 1.
I20260926 21:30:47.254305 7448 raft_consensus.cc:2454] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25878a5b4bad407f9592fdb2b6ce6d46 in term 1.
I20260926 21:30:47.254551 7527 leader_election.cc:302] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25878a5b4bad407f9592fdb2b6ce6d46, 36515a88910b4fd99cccc5f43c2ba661; no voters:
I20260926 21:30:47.254914 7881 raft_consensus.cc:2781] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:47.255352 7881 raft_consensus.cc:686] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 LEADER]: Becoming Leader. State: Replica: 25878a5b4bad407f9592fdb2b6ce6d46, State: Running, Role: LEADER
I20260926 21:30:47.255779 7881 consensus_queue.cc:237] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260926 21:30:47.256357 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.255894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.2:45603)
0926 21:30:47.256074 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.256203 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 21:30:47.259411 7314 catalog_manager.cc:5495] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25878a5b4bad407f9592fdb2b6ce6d46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } health_report { overall_health: UNKNOWN } } }
I20260926 21:30:47.277063 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.272282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.0.97.2:35983)
0926 21:30:47.272461 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.272466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.272499 (+ 33us) server_negotiation.cc:407] Connection header received
0926 21:30:47.272739 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.272743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.272814 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.272933 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:47.273221 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.273951 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.275298 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.275704 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.275788 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.276158 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:47.276641 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:47.276746 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.276827 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.276882 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:30:47.280789 7734 raft_consensus.cc:1261] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 1 FOLLOWER]: Refusing update from remote peer 25878a5b4bad407f9592fdb2b6ce6d46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:47.281049 7448 raft_consensus.cc:1261] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 1 FOLLOWER]: Refusing update from remote peer 25878a5b4bad407f9592fdb2b6ce6d46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:47.281535 7893 consensus_queue.cc:1035] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36515a88910b4fd99cccc5f43c2ba661" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:47.281900 7881 consensus_queue.cc:1035] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47732b6aa48c462cb58701cc6c40eb7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:47.284611 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.283917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.3:32929)
0926 21:30:47.284178 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.284333 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 21:30:47.285184 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.284598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.1:51741)
0926 21:30:47.284788 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.284924 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 21:30:47.287987 7896 mvcc.cc:205] Tried to move back new op lower bound from 7333716980855984128 to 7333716980764037120. Current Snapshot: MvccSnapshot[applied={T|T < 7333716980855984128}]
I20260926 21:30:47.290725 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.290259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.3:38433)
0926 21:30:47.290493 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.290583 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 21:30:47.291445 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.290251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.2:59237)
0926 21:30:47.291191 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.291293 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834}
I20260926 21:30:47.293335 7897 mvcc.cc:205] Tried to move back new op lower bound from 7333716980855984128 to 7333716980764037120. Current Snapshot: MvccSnapshot[applied={T|T < 7333716980855984128}]
I20260926 21:30:47.295656 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.294884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.2:53395)
0926 21:30:47.295403 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.295506 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
I20260926 21:30:47.297556 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.296943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.97.1:45853)
0926 21:30:47.297264 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.297368 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260926 21:30:47.304879 7314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:47.305142 7314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:47.310540 7314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3840b9c4dd1a40f1ba8be8abc9cfcbcd: Sending DeleteTablet for 3 replicas of tablet a45e2702fd2348bc9f3b9aba9931cd33
I20260926 21:30:47.311489 7573 tablet_service.cc:1514] Processing DeleteTablet for tablet a45e2702fd2348bc9f3b9aba9931cd33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:47 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:43084
I20260926 21:30:47.311581 7714 tablet_service.cc:1514] Processing DeleteTablet for tablet a45e2702fd2348bc9f3b9aba9931cd33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:47 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50232
I20260926 21:30:47.312232 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet a45e2702fd2348bc9f3b9aba9931cd33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:47 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57236
I20260926 21:30:47.318166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 7299
I20260926 21:30:47.323227 7907 tablet_replica.cc:330] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: stopping tablet replica
I20260926 21:30:47.323520 7907 raft_consensus.cc:2229] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:47.323917 7907 raft_consensus.cc:2258] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:47.324683 7906 tablet_replica.cc:330] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: stopping tablet replica
I20260926 21:30:47.324958 7906 raft_consensus.cc:2229] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:47.325314 7906 raft_consensus.cc:2258] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:47.326318 7907 ts_tablet_manager.cc:1905] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:47.330081 7905 tablet_replica.cc:330] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: stopping tablet replica
I20260926 21:30:47.330298 7905 raft_consensus.cc:2229] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:47.330968 7905 raft_consensus.cc:2258] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:47.331741 7906 ts_tablet_manager.cc:1905] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:47.337179 7906 ts_tablet_manager.cc:1918] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:47.337325 7906 log.cc:1196] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/a45e2702fd2348bc9f3b9aba9931cd33
I20260926 21:30:47.337608 7905 ts_tablet_manager.cc:1905] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:47.337759 7906 ts_tablet_manager.cc:1939] T a45e2702fd2348bc9f3b9aba9931cd33 P 36515a88910b4fd99cccc5f43c2ba661: Deleting consensus metadata
I20260926 21:30:47.338943 7298 catalog_manager.cc:4841] TS 36515a88910b4fd99cccc5f43c2ba661 (127.0.97.3:36471): tablet a45e2702fd2348bc9f3b9aba9931cd33 (table test-table [id=d8d3fdce7a7a4402b8a1f59c0a03bd98]) successfully deleted
I20260926 21:30:47.339500 7907 ts_tablet_manager.cc:1918] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:47.339659 7907 log.cc:1196] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/a45e2702fd2348bc9f3b9aba9931cd33
I20260926 21:30:47.340135 7907 ts_tablet_manager.cc:1939] T a45e2702fd2348bc9f3b9aba9931cd33 P 47732b6aa48c462cb58701cc6c40eb7d: Deleting consensus metadata
I20260926 21:30:47.341251 7297 catalog_manager.cc:4841] TS 47732b6aa48c462cb58701cc6c40eb7d (127.0.97.1:35291): tablet a45e2702fd2348bc9f3b9aba9931cd33 (table test-table [id=d8d3fdce7a7a4402b8a1f59c0a03bd98]) successfully deleted
I20260926 21:30:47.346318 7905 ts_tablet_manager.cc:1918] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:47.346622 7905 log.cc:1196] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/a45e2702fd2348bc9f3b9aba9931cd33
I20260926 21:30:47.347112 7905 ts_tablet_manager.cc:1939] T a45e2702fd2348bc9f3b9aba9931cd33 P 25878a5b4bad407f9592fdb2b6ce6d46: Deleting consensus metadata
I20260926 21:30:47.348308 7297 catalog_manager.cc:4841] TS 25878a5b4bad407f9592fdb2b6ce6d46 (127.0.97.2:35983): tablet a45e2702fd2348bc9f3b9aba9931cd33 (table test-table [id=d8d3fdce7a7a4402b8a1f59c0a03bd98]) successfully deleted
I20260926 21:30:47.351258 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.319109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.0.97.1:35291)
0926 21:30:47.319386 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.319395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.334613 (+ 15218us) server_negotiation.cc:407] Connection header received
0926 21:30:47.334821 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.334825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.334903 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.334998 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:47.336722 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.337856 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.339847 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.340568 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.340712 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.344973 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:47.344994 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:47.345016 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:47.345055 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:47.348457 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:47.349156 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:47.349165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:47.349169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:47.349295 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:47.349672 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:47.349677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:47.349680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:47.350082 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:47.350377 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.350709 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.350916 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 21:30:47.351940 7488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:47.469779 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 7299
I20260926 21:30:47.492471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 7505
I20260926 21:30:47.513953 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.493313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.0.97.2:35983)
0926 21:30:47.493526 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.493533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.493565 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:30:47.493703 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.493710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.493796 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.493930 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:47.495186 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.496436 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.497853 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.498567 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.498716 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.506733 (+ 8017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:47.506749 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:47.506769 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:47.506817 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:47.510792 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:47.511526 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:47.511536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:47.511540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:47.511784 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:47.512174 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:47.512179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:47.512183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:47.512674 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:47.513040 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.513216 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.513481 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 21:30:47.514842 7633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:47.631282 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.630827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54186)
0926 21:30:47.630994 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.631114 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:30:47.632179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 7505
I20260926 21:30:47.632368 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.631943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54190)
0926 21:30:47.632150 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:47.632229 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 21:30:47.651435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 7651
I20260926 21:30:47.665194 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:47.652139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.0.97.3:36471)
0926 21:30:47.652311 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:47.652315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:47.652333 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:47.652479 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:47.652483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:47.652543 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:47.652640 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:47.654263 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.655065 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.656266 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:47.656671 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:47.656808 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:47.659365 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:47.659380 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:47.659395 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:47.659441 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:47.662355 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:47.663115 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:47.663126 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:47.663136 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:47.663259 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:47.663676 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:47.663680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:47.663682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:47.664035 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:47.664324 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:47.664659 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 21:30:47.664874 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:30:47.666110 7774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:47.761061 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 7651
I20260926 21:30:47.779543 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 6990
I20260926 21:30:47.780612 7157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:47.894929 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 6990
I20260926 21:30:47.915127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 7086
I20260926 21:30:47.916184 7344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:48.032565 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:48.032176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37433 (local address 127.0.0.1:54204)
0926 21:30:48.032325 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:48.032414 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 21:30:48.032605 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 7086
2026-09-26T21:30:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9441 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:48 dist-test-slave-87bt krb5kdc[7916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:48 dist-test-slave-87bt krb5kdc[7916](info): set up 2 sockets
Sep 26 21:30:48 dist-test-slave-87bt krb5kdc[7916](info): commencing operation
krb5kdc: starting...
W20260926 21:30:49.131418 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:49 dist-test-slave-87bt krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 21:30:49.346048 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37005
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:49.672418 7929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:49.672719 7929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:49.672915 7929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:49.680264 7929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:49.680369 7929 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 21:30:49.680435 7929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:49.680524 7929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:49.680588 7929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:49.680716 7929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:49.690456 7929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37005
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:49.691742 7929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:49.693121 7929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:49.704468 7934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:49.704556 7935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:49.704818 7937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:49.705694 7929 server_base.cc:1034] running on GCE node
I20260926 21:30:49.706528 7929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:49.707890 7929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:49.709146 7929 hybrid_clock.cc:648] HybridClock initialized: now 1790458249709093 us; error 46 us; skew 500 ppm
I20260926 21:30:49.709493 7929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:49 dist-test-slave-87bt krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:49.712499 7929 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 21:30:49.713935 7929 webserver.cc:460] Webserver started at http://127.0.97.62:34203/ using document root <none> and password file <none>
I20260926 21:30:49.714668 7929 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:49.714792 7929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:49.715111 7929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:49.717936 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "c77c5e52ef804fc18cad445e8b0ce4df"
format_stamp: "Formatted at 2026-09-26 21:30:49 on dist-test-slave-87bt"
server_key: "01f152d8c5837854e55966bbe5eecf7e"
server_key_iv: "0439425fb3d8396f48bf164548355d5c"
server_key_version: "encryptionkey@0"
I20260926 21:30:49.718809 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "c77c5e52ef804fc18cad445e8b0ce4df"
format_stamp: "Formatted at 2026-09-26 21:30:49 on dist-test-slave-87bt"
server_key: "01f152d8c5837854e55966bbe5eecf7e"
server_key_iv: "0439425fb3d8396f48bf164548355d5c"
server_key_version: "encryptionkey@0"
I20260926 21:30:49.723654 7929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 21:30:49.726985 7944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:49.728271 7929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:30:49.728398 7929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "c77c5e52ef804fc18cad445e8b0ce4df"
format_stamp: "Formatted at 2026-09-26 21:30:49 on dist-test-slave-87bt"
server_key: "01f152d8c5837854e55966bbe5eecf7e"
server_key_iv: "0439425fb3d8396f48bf164548355d5c"
server_key_version: "encryptionkey@0"
I20260926 21:30:49.728565 7929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:49.762570 7929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:49.766479 7929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:49.766709 7929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:49.787662 7929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37005
I20260926 21:30:49.787686 8005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37005 every 8 connection(s)
I20260926 21:30:49.789052 7929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:49.792637 8006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:49.796774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 7929
I20260926 21:30:49.796965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:49.797744 388 external_mini_cluster.cc:1428] Setting key 2bdb78f2efa9527ecf734c91cfc4e554
I20260926 21:30:49.803762 8006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: Bootstrap starting.
I20260926 21:30:49.806759 8006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:49.807750 8006 log.cc:824] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:49.810483 8006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: No bootstrap required, opened a new log
Sep 26 21:30:49 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:49.818470 8006 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77c5e52ef804fc18cad445e8b0ce4df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37005 } }
I20260926 21:30:49.818884 8006 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:49.818996 8006 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77c5e52ef804fc18cad445e8b0ce4df, State: Initialized, Role: FOLLOWER
I20260926 21:30:49.819592 8006 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77c5e52ef804fc18cad445e8b0ce4df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37005 } }
I20260926 21:30:49.819914 8006 raft_consensus.cc:388] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:49.819937 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:49.799754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.0.97.62:37005)
0926 21:30:49.800455 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:49.800466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:49.800506 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:30:49.802530 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:49.802555 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:49.802962 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:49.803343 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:49.804753 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:49.806137 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:49.807512 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:49.808324 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:49.808526 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:49.812511 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:49.812521 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:49.812536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:49.812576 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:49.815617 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:49.816294 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:49.816304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:49.816310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:49.816477 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:49.816854 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:49.816863 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:49.816866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:49.817405 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:49.817748 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:49.818118 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 21:30:49.818580 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":301,"threads_started":1}
I20260926 21:30:49.820075 8006 raft_consensus.cc:482] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:49.820307 8006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:49.822677 8006 raft_consensus.cc:504] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77c5e52ef804fc18cad445e8b0ce4df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37005 } }
I20260926 21:30:49.823239 8006 leader_election.cc:302] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c77c5e52ef804fc18cad445e8b0ce4df; no voters:
I20260926 21:30:49.823765 8006 leader_election.cc:288] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:49.823935 8011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:49.824609 8011 raft_consensus.cc:686] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [term 1 LEADER]: Becoming Leader. State: Replica: c77c5e52ef804fc18cad445e8b0ce4df, State: Running, Role: LEADER
I20260926 21:30:49.825141 8011 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77c5e52ef804fc18cad445e8b0ce4df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37005 } }
I20260926 21:30:49.825799 8006 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:49.828174 8013 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77c5e52ef804fc18cad445e8b0ce4df. Latest consensus state: current_term: 1 leader_uuid: "c77c5e52ef804fc18cad445e8b0ce4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77c5e52ef804fc18cad445e8b0ce4df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37005 } } }
I20260926 21:30:49.828527 8013 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:49.829207 8012 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c77c5e52ef804fc18cad445e8b0ce4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77c5e52ef804fc18cad445e8b0ce4df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37005 } } }
I20260926 21:30:49.829582 8012 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:49.831645 8019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:49.836942 8019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:49.845542 8019 catalog_manager.cc:1347] Generated new cluster ID: 35cc357e2b7f471ea4bd553159f0db85
I20260926 21:30:49.845664 8019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:49.853281 8019 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:49.854637 8019 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:49.861933 8019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: Generated new TSK 0
I20260926 21:30:49.862967 8019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 21:30:49.919127 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37005
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:50.245256 8032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:50.245606 8032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:50.245774 8032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:50.253091 8032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:50.253259 8032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:50.253407 8032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:30:50.253540 8032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:50.263479 8032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:50.264801 8032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:50.266150 8032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:50.279462 8037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:50.279500 8038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:50.279539 8040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:50.280435 8032 server_base.cc:1034] running on GCE node
I20260926 21:30:50.281108 8032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:50.282048 8032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:50.283308 8032 hybrid_clock.cc:648] HybridClock initialized: now 1790458250283224 us; error 53 us; skew 500 ppm
I20260926 21:30:50.283617 8032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:50 dist-test-slave-87bt krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:50.286688 8032 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 21:30:50.288132 8032 webserver.cc:460] Webserver started at http://127.0.97.1:44831/ using document root <none> and password file <none>
I20260926 21:30:50.288848 8032 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:50.288980 8032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:50.289300 8032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:50.292228 8032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "49574611798d4fc2a9b902ea838e9ce7"
format_stamp: "Formatted at 2026-09-26 21:30:50 on dist-test-slave-87bt"
server_key: "086ba84caffb6367aa9b8f4acd47f77b"
server_key_iv: "bf4bc5fb0597c7955a04d6407a84cdf6"
server_key_version: "encryptionkey@0"
I20260926 21:30:50.293026 8032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "49574611798d4fc2a9b902ea838e9ce7"
format_stamp: "Formatted at 2026-09-26 21:30:50 on dist-test-slave-87bt"
server_key: "086ba84caffb6367aa9b8f4acd47f77b"
server_key_iv: "bf4bc5fb0597c7955a04d6407a84cdf6"
server_key_version: "encryptionkey@0"
I20260926 21:30:50.298041 8032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:30:50.301344 8047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:50.302886 8032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:50.303078 8032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "49574611798d4fc2a9b902ea838e9ce7"
format_stamp: "Formatted at 2026-09-26 21:30:50 on dist-test-slave-87bt"
server_key: "086ba84caffb6367aa9b8f4acd47f77b"
server_key_iv: "bf4bc5fb0597c7955a04d6407a84cdf6"
server_key_version: "encryptionkey@0"
I20260926 21:30:50.303318 8032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:50.334117 8032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:50.338300 8032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:50.338636 8032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:50.341261 8032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:50.341378 8032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:50.341548 8032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:50.341651 8032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:30:50 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:50.365624 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:50.341857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37471 (local address 127.0.97.62:37005)
0926 21:30:50.342733 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:50.342741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:50.343856 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 21:30:50.345813 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:50.345819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:50.345918 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:50.346077 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:50.348320 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.349652 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.352251 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.352844 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.352988 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:50.357841 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:50.357852 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:50.357861 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:50.357924 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:50.362056 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:50.363008 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:50.363017 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:50.363022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:50.363223 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:50.363793 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:50.363799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:50.363802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:50.364084 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:50.364243 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:50.365051 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 21:30:50.365270 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":643,"threads_started":1}
I20260926 21:30:50.366961 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:50.342768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37005 (local address 127.0.97.1:37471)
0926 21:30:50.343661 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:50.343717 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:50.345528 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:50.346378 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:50.346388 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:50.347227 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:50.348095 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:50.348117 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.349887 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.349896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:50.352074 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:50.352099 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.353106 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.353117 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:50.353468 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:50.354305 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:50.354354 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:50.357517 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:50.362227 (+ 4710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:50.362239 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:50.362264 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:50.362823 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:50.363383 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:50.363387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:50.363391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:50.363657 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:50.364223 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:50.364231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:50.364680 (+ 449us) client_negotiation.cc:769] Sending connection context
0926 21:30:50.364960 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 21:30:50.365512 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":386,"threads_started":1}
I20260926 21:30:50.389074 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:50.373370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49833 (local address 127.0.97.62:37005)
0926 21:30:50.373527 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:50.373531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:50.374468 (+ 937us) server_negotiation.cc:407] Connection header received
0926 21:30:50.374667 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:50.374673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:50.374743 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:50.374874 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:50.376548 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.377490 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.380109 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.380496 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.380591 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:50.383396 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:50.383404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:50.383408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:50.383452 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:50.386489 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:50.387158 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:50.387163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:50.387167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:50.387297 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:50.387688 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:50.387694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:50.387697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:50.387952 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:50.388104 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:50.388512 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 21:30:50.388724 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:30:50.389191 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:50.373702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37005 (local address 127.0.97.1:49833)
0926 21:30:50.374314 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:50.374347 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:50.374532 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:50.374920 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:50.374928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:50.375720 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:50.376349 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:50.376369 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.377684 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.377693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:50.379943 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:50.379966 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.380704 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.380714 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:50.380950 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:50.381667 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:50.381698 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:50.383152 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:50.386617 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:50.386623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:50.386627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:50.387030 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:50.387437 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:50.387442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:50.387446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:50.387577 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:50.388068 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:50.388073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:50.388260 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 21:30:50.388397 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 21:30:50.388723 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":367,"threads_started":1}
I20260926 21:30:50.396138 8032 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44861
I20260926 21:30:50.396159 8167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44861 every 8 connection(s)
I20260926 21:30:50.397347 8032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:50.399283 8168 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37005
I20260926 21:30:50.399513 8168 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:50.400139 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 8032
I20260926 21:30:50.400162 8168 heartbeater.cc:511] Master 127.0.97.62:37005 requested a full tablet report, sending...
I20260926 21:30:50.400349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:50.401031 388 external_mini_cluster.cc:1428] Setting key 2241826685d1494d80b1a560e76ddd51
I20260926 21:30:50.402474 7961 ts_manager.cc:194] Registered new tserver with Master: 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861)
I20260926 21:30:50.403998 7961 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37471
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 21:30:50.445564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37005
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:50.774238 8172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:50.774617 8172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:50.774801 8172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:50.782051 8172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:50.782225 8172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:50.782394 8172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:30:50.782565 8172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:50.792514 8172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:50.793826 8172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:50.795240 8172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:50.808569 8180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:50.808552 8178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:50.809414 8177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:50.809365 8172 server_base.cc:1034] running on GCE node
I20260926 21:30:50.810180 8172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:50.811117 8172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:50.812372 8172 hybrid_clock.cc:648] HybridClock initialized: now 1790458250812300 us; error 46 us; skew 500 ppm
I20260926 21:30:50.812682 8172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:50 dist-test-slave-87bt krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:50.815730 8172 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 21:30:50.817159 8172 webserver.cc:460] Webserver started at http://127.0.97.2:40573/ using document root <none> and password file <none>
I20260926 21:30:50.817855 8172 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:50.818012 8172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:50.818351 8172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:50.821267 8172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "5099d51479c64cc9a3575f1ef3103e80"
format_stamp: "Formatted at 2026-09-26 21:30:50 on dist-test-slave-87bt"
server_key: "b48549547893b3876b3a0b6827d2c278"
server_key_iv: "cba6f5da113e13963408f81f7fb7348e"
server_key_version: "encryptionkey@0"
I20260926 21:30:50.822049 8172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5099d51479c64cc9a3575f1ef3103e80"
format_stamp: "Formatted at 2026-09-26 21:30:50 on dist-test-slave-87bt"
server_key: "b48549547893b3876b3a0b6827d2c278"
server_key_iv: "cba6f5da113e13963408f81f7fb7348e"
server_key_version: "encryptionkey@0"
I20260926 21:30:50.827158 8172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 21:30:50.830711 8187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:50.832794 8172 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 21:30:50.832981 8172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "5099d51479c64cc9a3575f1ef3103e80"
format_stamp: "Formatted at 2026-09-26 21:30:50 on dist-test-slave-87bt"
server_key: "b48549547893b3876b3a0b6827d2c278"
server_key_iv: "cba6f5da113e13963408f81f7fb7348e"
server_key_version: "encryptionkey@0"
I20260926 21:30:50.833264 8172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:50.864333 8172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:50.868531 8172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:50.868825 8172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:50.871734 8172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:50.871845 8172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:50.872015 8172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:50.872094 8172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:30:50 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:50.896116 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:50.872149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60003 (local address 127.0.97.62:37005)
0926 21:30:50.872400 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:50.872410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:50.873958 (+ 1548us) server_negotiation.cc:407] Connection header received
0926 21:30:50.876140 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:50.876149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:50.876264 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:50.876398 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:50.879862 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.880801 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.883498 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.884297 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.884458 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:50.888748 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:50.888756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:50.888762 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:50.888806 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:50.892202 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:50.893306 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:50.893312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:50.893316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:50.893484 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:50.894085 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:50.894097 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:50.894099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:50.894552 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:50.894716 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:50.895545 (+ 829us) server_negotiation.cc:299] Negotiation successful
0926 21:30:50.895759 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 21:30:50.897183 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:50.872905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37005 (local address 127.0.97.2:60003)
0926 21:30:50.873746 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:50.873813 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:50.875849 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:50.876881 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:50.876898 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:50.877938 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:50.879589 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:50.879641 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.881047 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.881058 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:50.883252 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:50.883331 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.884607 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.884617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:50.884812 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:50.885512 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:50.885535 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:50.888335 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:50.892365 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:50.892377 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:50.892398 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:50.893124 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:50.893647 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:50.893654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:50.893660 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:50.893946 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:50.894694 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:50.894706 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:50.895163 (+ 457us) client_negotiation.cc:769] Sending connection context
0926 21:30:50.895452 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 21:30:50.896017 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":358,"threads_started":1}
I20260926 21:30:50.918848 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:50.903027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37005 (local address 127.0.97.2:51007)
0926 21:30:50.903710 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:50.903737 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:50.903927 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:50.904269 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:50.904276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:50.904977 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:50.905451 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:50.905467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.906472 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.906481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:50.908806 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:50.908827 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.909694 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.909703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:50.909985 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:50.910907 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:50.910940 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:50.912054 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:50.915216 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:50.915221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:50.915225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:50.915825 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:50.916360 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:50.916368 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:50.916374 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:50.916571 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:50.917264 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:50.917272 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:50.917600 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 21:30:50.917813 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 21:30:50.918316 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":331,"threads_started":1}
I20260926 21:30:50.919062 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:50.902677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51007 (local address 127.0.97.62:37005)
0926 21:30:50.902874 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:50.902882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:50.903852 (+ 970us) server_negotiation.cc:407] Connection header received
0926 21:30:50.904054 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:50.904059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:50.904119 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:50.904200 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:50.905620 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.906321 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.908972 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:50.909468 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:50.909610 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:50.912255 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:50.912261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:50.912266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:50.912310 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:50.915057 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:50.915963 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:50.915971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:50.915976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:50.916129 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:50.916727 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:50.916734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:50.916738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:50.917116 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:50.917308 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:50.918513 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0926 21:30:50.918741 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:30:50.929840 8172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43587
I20260926 21:30:50.929872 8306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43587 every 8 connection(s)
I20260926 21:30:50.931222 8172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:50.933487 8307 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37005
I20260926 21:30:50.933697 8307 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:50.934304 8307 heartbeater.cc:511] Master 127.0.97.62:37005 requested a full tablet report, sending...
I20260926 21:30:50.935551 7961 ts_manager.cc:194] Registered new tserver with Master: 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587)
I20260926 21:30:50.936299 7961 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60003
I20260926 21:30:50.936887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 8172
I20260926 21:30:50.937053 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:50.937460 388 external_mini_cluster.cc:1428] Setting key 9eaf637e52b999ad411021420df8e852
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 21:30:50.983006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37005
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:51.305907 8311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:51.306190 8311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:51.306288 8311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:51.313711 8311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:51.313817 8311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:51.313898 8311 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:30:51.313969 8311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:51.323725 8311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:51.324914 8311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:51.326336 8311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:51.339217 8316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:51.339218 8317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:51.339221 8319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:51.340520 8311 server_base.cc:1034] running on GCE node
I20260926 21:30:51.341238 8311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:51.342118 8311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:51.343390 8311 hybrid_clock.cc:648] HybridClock initialized: now 1790458251343312 us; error 63 us; skew 500 ppm
I20260926 21:30:51.343683 8311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:51 dist-test-slave-87bt krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:51.346787 8311 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 21:30:51.348165 8311 webserver.cc:460] Webserver started at http://127.0.97.3:40977/ using document root <none> and password file <none>
I20260926 21:30:51.348829 8311 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:51.348955 8311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:51.349268 8311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:51.352186 8311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "53621701c7d544b18cc974524b4362d7"
format_stamp: "Formatted at 2026-09-26 21:30:51 on dist-test-slave-87bt"
server_key: "2e03ba20b872dae8c17c73adcb2ac70d"
server_key_iv: "838bf73e86c4d30c5ac066eb5e8d6370"
server_key_version: "encryptionkey@0"
I20260926 21:30:51.352982 8311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "53621701c7d544b18cc974524b4362d7"
format_stamp: "Formatted at 2026-09-26 21:30:51 on dist-test-slave-87bt"
server_key: "2e03ba20b872dae8c17c73adcb2ac70d"
server_key_iv: "838bf73e86c4d30c5ac066eb5e8d6370"
server_key_version: "encryptionkey@0"
I20260926 21:30:51.358292 8311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:30:51.361549 8326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:51.362859 8311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:51.363034 8311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "53621701c7d544b18cc974524b4362d7"
format_stamp: "Formatted at 2026-09-26 21:30:51 on dist-test-slave-87bt"
server_key: "2e03ba20b872dae8c17c73adcb2ac70d"
server_key_iv: "838bf73e86c4d30c5ac066eb5e8d6370"
server_key_version: "encryptionkey@0"
I20260926 21:30:51.363260 8311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:51.377260 8311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:51.381234 8311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:51.381544 8311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:51.384298 8311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:51.384430 8311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:51.384598 8311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:51.384689 8311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:30:51 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:51.407232 8168 heartbeater.cc:503] Master 127.0.97.62:37005 was elected leader, sending a full tablet report...
I20260926 21:30:51.408620 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.384809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51389 (local address 127.0.97.62:37005)
0926 21:30:51.385068 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.385080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.386466 (+ 1386us) server_negotiation.cc:407] Connection header received
0926 21:30:51.388465 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.388471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.388560 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.388671 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:51.390736 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.392152 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.394671 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.395241 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.395414 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.400769 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:51.400776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:51.400783 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:51.400826 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:51.404653 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:51.405799 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:51.405804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:51.405808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:51.405937 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:51.406551 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:51.406558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:51.406563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:51.406859 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:51.407017 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.408031 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.408263 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:30:51.409981 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.385462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37005 (local address 127.0.97.3:51389)
0926 21:30:51.386277 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:51.386324 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:51.388191 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:51.388903 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:51.388914 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:51.389645 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:51.390503 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.390534 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.392377 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.392385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.394498 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.394513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.395710 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.395723 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.396014 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.396931 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:51.396962 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:51.400460 (+ 3498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:51.404853 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:51.404865 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:51.404886 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:51.405652 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:51.406061 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:51.406069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:51.406075 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:51.406382 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:51.406994 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:51.407002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:51.407458 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 21:30:51.407806 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 21:30:51.408307 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":370,"threads_started":1}
I20260926 21:30:51.430687 8311 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46561
I20260926 21:30:51.430751 8445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46561 every 8 connection(s)
I20260926 21:30:51.432129 8311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:51.432950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 8311
I20260926 21:30:51.433167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:51.433862 388 external_mini_cluster.cc:1428] Setting key 0429900a9258f0c2eb565987e100ed27
I20260926 21:30:51.434284 8446 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37005
I20260926 21:30:51.434533 8446 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:51.435407 8446 heartbeater.cc:511] Master 127.0.97.62:37005 requested a full tablet report, sending...
I20260926 21:30:51.436061 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.417149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59603 (local address 127.0.97.62:37005)
0926 21:30:51.417367 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.417373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.419621 (+ 2248us) server_negotiation.cc:407] Connection header received
0926 21:30:51.419824 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.419831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.419923 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.420041 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:51.421631 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.422832 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.425598 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.426355 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.426560 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.429309 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:51.429318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:51.429325 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:51.429367 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:51.432748 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:51.433701 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:51.433706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:51.433710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:51.433865 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:51.434297 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:51.434305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:51.434310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:51.434707 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:51.434826 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.435473 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.435716 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:30:51.436910 7960 ts_manager.cc:194] Registered new tserver with Master: 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:51.437033 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.417427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37005 (local address 127.0.97.3:59603)
0926 21:30:51.419471 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:51.419497 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:51.419675 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:51.420085 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:51.420097 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:51.420850 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:51.421419 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.421445 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.423068 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.423075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.425332 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.425355 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.426632 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.426639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.426929 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.427890 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:51.427917 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:51.429091 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:51.433002 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:51.433006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:51.433013 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:51.433565 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:51.434013 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:51.434017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:51.434020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:51.434167 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:51.434922 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:51.434928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:51.435156 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 21:30:51.436221 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0926 21:30:51.436568 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1848,"mutex_wait_us":4,"thread_start_us":1630,"threads_started":1}
I20260926 21:30:51.437875 7960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51389
I20260926 21:30:51.449728 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:30:51 dist-test-slave-87bt krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458251, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:30:51 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:30:51 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 21:30:51.497165 8056 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:51.486995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.0.97.62:37005)
0926 21:30:51.487217 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.487224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.487477 (+ 253us) server_negotiation.cc:407] Connection header received
0926 21:30:51.487682 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.487687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.487768 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.487877 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:51.489148 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.489865 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.491194 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.491621 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.491770 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.496874 (+ 5104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47356: BlockingRecv error: recv got EOF from 127.0.0.1:47356 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 21:30:51 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458251, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:51.512588 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.500207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.0.97.62:37005)
0926 21:30:51.500361 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.500365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.500650 (+ 285us) server_negotiation.cc:407] Connection header received
0926 21:30:51.500773 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.500777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.500829 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.500903 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:51.501957 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.502698 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.503849 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.504250 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.504344 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.507956 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:51.507961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:51.507964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:51.507993 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:51.510526 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:51.511054 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:51.511058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:51.511060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:51.511167 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:51.511500 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:51.511504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:51.511506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:51.511748 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:51.511945 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.512180 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.512329 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:30:51.516350 7960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:51.523336 7960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:51.544471 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.535398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43587 (local address 127.0.0.1:35102)
0926 21:30:51.536210 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:51.536252 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:51.536403 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:51.537591 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:51.537602 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:51.537639 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:51.538795 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.538805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.540508 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.540516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.541952 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.541964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.543565 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.543571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.543792 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.543827 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:30:51.543960 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 21:30:51.544145 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":427,"threads_started":1}
I20260926 21:30:51.544454 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.535585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.0.97.2:43587)
0926 21:30:51.537076 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.537084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.537112 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:30:51.537198 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.537203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.537396 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.537760 (+ 364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:51.538984 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.540387 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.542097 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.543354 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.543504 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.543710 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.544007 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.544160 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":1104,"threads_started":1}
I20260926 21:30:51.548184 8237 tablet_service.cc:1467] Processing CreateTablet for tablet c151fa447a0e4932a02b27e75c84d23a (DEFAULT_TABLE table=test-table [id=b2d4078300f74d85b8dc44f7be6e8320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:51.549705 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c151fa447a0e4932a02b27e75c84d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:51.554713 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.535510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44861 (local address 127.0.0.1:42648)
0926 21:30:51.538663 (+ 3153us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:51.538686 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:51.538802 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:51.547928 (+ 9126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:51.547932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:51.547957 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:51.548325 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.548337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.550091 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.550097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.552126 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.552140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.553926 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.553932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.554088 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.554122 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:30:51.554216 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 21:30:51.554349 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2877,"thread_start_us":14423,"threads_started":1}
I20260926 21:30:51.554773 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.536060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.0.97.1:44861)
0926 21:30:51.547257 (+ 11197us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.547284 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.547320 (+ 36us) server_negotiation.cc:407] Connection header received
0926 21:30:51.547434 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.547444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.547711 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.547933 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:51.548512 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.549934 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.552315 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.553715 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.553857 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.554052 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.554275 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.554479 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11008,"thread_start_us":13540,"threads_started":1}
I20260926 21:30:51.557648 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.537892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46561 (local address 127.0.0.1:59832)
0926 21:30:51.545068 (+ 7176us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:51.545084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:51.545167 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:51.551714 (+ 6547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:51.551718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:51.551747 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:51.551980 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.551992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.553431 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.553438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.555229 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.555244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.556985 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.556992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.557168 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.557199 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:30:51.557285 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 21:30:51.557409 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7093,"thread_start_us":350,"threads_started":1}
I20260926 21:30:51.557791 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.538128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.0.97.3:46561)
0926 21:30:51.548672 (+ 10544us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.548683 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.548734 (+ 51us) server_negotiation.cc:407] Connection header received
0926 21:30:51.548843 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.548853 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.549094 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.549327 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:51.552198 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.553293 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.555396 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.556819 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.556945 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.557149 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.557356 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.557500 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10341,"thread_start_us":10159,"threads_started":1}
I20260926 21:30:51.558205 8098 tablet_service.cc:1467] Processing CreateTablet for tablet c151fa447a0e4932a02b27e75c84d23a (DEFAULT_TABLE table=test-table [id=b2d4078300f74d85b8dc44f7be6e8320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:51.559593 8098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c151fa447a0e4932a02b27e75c84d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:51.561364 8376 tablet_service.cc:1467] Processing CreateTablet for tablet c151fa447a0e4932a02b27e75c84d23a (DEFAULT_TABLE table=test-table [id=b2d4078300f74d85b8dc44f7be6e8320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:51.562716 8376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c151fa447a0e4932a02b27e75c84d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:51.564112 8470 tablet_bootstrap.cc:492] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Bootstrap starting.
I20260926 21:30:51.567584 8470 tablet_bootstrap.cc:654] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:51.568917 8470 log.cc:824] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:51.571889 8470 tablet_bootstrap.cc:492] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: No bootstrap required, opened a new log
I20260926 21:30:51.572342 8470 ts_tablet_manager.cc:1397] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 21:30:51.572359 8472 tablet_bootstrap.cc:492] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Bootstrap starting.
I20260926 21:30:51.575788 8473 tablet_bootstrap.cc:492] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Bootstrap starting.
I20260926 21:30:51.576426 8472 tablet_bootstrap.cc:654] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:51.577382 8472 log.cc:824] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:51.579164 8473 tablet_bootstrap.cc:654] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:51.579969 8472 tablet_bootstrap.cc:492] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: No bootstrap required, opened a new log
I20260926 21:30:51.580229 8472 ts_tablet_manager.cc:1397] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 21:30:51.580345 8473 log.cc:824] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:51.582551 8470 raft_consensus.cc:348] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.583019 8470 raft_consensus.cc:374] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:51.583132 8470 raft_consensus.cc:729] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5099d51479c64cc9a3575f1ef3103e80, State: Initialized, Role: FOLLOWER
I20260926 21:30:51.583511 8473 tablet_bootstrap.cc:492] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: No bootstrap required, opened a new log
I20260926 21:30:51.583766 8473 ts_tablet_manager.cc:1397] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260926 21:30:51.583765 8470 consensus_queue.cc:260] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.584802 8307 heartbeater.cc:503] Master 127.0.97.62:37005 was elected leader, sending a full tablet report...
I20260926 21:30:51.585436 8470 ts_tablet_manager.cc:1428] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 21:30:51.589790 8472 raft_consensus.cc:348] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.590200 8472 raft_consensus.cc:374] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:51.590294 8472 raft_consensus.cc:729] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49574611798d4fc2a9b902ea838e9ce7, State: Initialized, Role: FOLLOWER
I20260926 21:30:51.590974 8472 consensus_queue.cc:260] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.592590 8472 ts_tablet_manager.cc:1428] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 21:30:51.593365 8473 raft_consensus.cc:348] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.593806 8473 raft_consensus.cc:374] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:51.593920 8473 raft_consensus.cc:729] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53621701c7d544b18cc974524b4362d7, State: Initialized, Role: FOLLOWER
I20260926 21:30:51.594578 8473 consensus_queue.cc:260] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.595814 8446 heartbeater.cc:503] Master 127.0.97.62:37005 was elected leader, sending a full tablet report...
I20260926 21:30:51.596402 8473 ts_tablet_manager.cc:1428] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260926 21:30:51.649734 8169 tablet.cc:2369] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:51.657686 8476 raft_consensus.cc:482] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:51.658043 8476 raft_consensus.cc:504] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.659782 8476 leader_election.cc:288] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861), 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:51.669073 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.660219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33911 (local address 127.0.97.1:44861)
0926 21:30:51.660423 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.660431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.661138 (+ 707us) server_negotiation.cc:407] Connection header received
0926 21:30:51.661243 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.661248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.661328 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.661430 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:51.662314 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.663413 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.666472 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.667718 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.667872 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.667987 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.668649 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.668785 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:30:51.669113 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.660294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56043 (local address 127.0.97.3:46561)
0926 21:30:51.660608 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.660616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.661008 (+ 392us) server_negotiation.cc:407] Connection header received
0926 21:30:51.661244 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.661250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.661328 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.661430 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:51.662275 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.663413 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.666468 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.667779 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.667889 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.667983 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.668649 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.668784 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260926 21:30:51.669247 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.660124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44861 (local address 127.0.97.2:33911)
0926 21:30:51.661033 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:51.661053 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:51.661153 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:51.661460 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:51.661465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:51.661490 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:51.662167 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.662179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.663573 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.663576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.666283 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.666298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.668055 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.668060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.668325 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.668476 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 21:30:51.668578 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 21:30:51.668771 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":334,"thread_start_us":650,"threads_started":1}
I20260926 21:30:51.669586 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.660047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46561 (local address 127.0.97.2:56043)
0926 21:30:51.660900 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:51.660928 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:51.661098 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:51.661460 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:51.661465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:51.661546 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:51.662096 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.662116 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.663566 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.663572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.666282 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:51.666298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.668053 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.668060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:51.668323 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.668479 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 21:30:51.668578 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 21:30:51.668771 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":320,"spinlock_wait_cycles":23808,"thread_start_us":447,"threads_started":1}
I20260926 21:30:51.670409 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c151fa447a0e4932a02b27e75c84d23a" candidate_uuid: "5099d51479c64cc9a3575f1ef3103e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49574611798d4fc2a9b902ea838e9ce7" is_pre_election: true
I20260926 21:30:51.670878 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c151fa447a0e4932a02b27e75c84d23a" candidate_uuid: "5099d51479c64cc9a3575f1ef3103e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53621701c7d544b18cc974524b4362d7" is_pre_election: true
I20260926 21:30:51.671048 8123 raft_consensus.cc:2454] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5099d51479c64cc9a3575f1ef3103e80 in term 0.
I20260926 21:30:51.671406 8398 raft_consensus.cc:2454] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5099d51479c64cc9a3575f1ef3103e80 in term 0.
I20260926 21:30:51.671788 8188 leader_election.cc:302] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 5099d51479c64cc9a3575f1ef3103e80; no voters:
I20260926 21:30:51.672328 8476 raft_consensus.cc:2781] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:51.672498 8476 raft_consensus.cc:482] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:51.672603 8476 raft_consensus.cc:3037] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:51.674939 8476 raft_consensus.cc:504] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.675535 8476 leader_election.cc:288] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [CANDIDATE]: Term 1 election: Requested vote from peers 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861), 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:51.676041 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c151fa447a0e4932a02b27e75c84d23a" candidate_uuid: "5099d51479c64cc9a3575f1ef3103e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49574611798d4fc2a9b902ea838e9ce7"
I20260926 21:30:51.676036 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c151fa447a0e4932a02b27e75c84d23a" candidate_uuid: "5099d51479c64cc9a3575f1ef3103e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53621701c7d544b18cc974524b4362d7"
I20260926 21:30:51.676237 8398 raft_consensus.cc:3037] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:51.676247 8123 raft_consensus.cc:3037] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:51.679427 8398 raft_consensus.cc:2454] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5099d51479c64cc9a3575f1ef3103e80 in term 1.
I20260926 21:30:51.679425 8123 raft_consensus.cc:2454] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5099d51479c64cc9a3575f1ef3103e80 in term 1.
I20260926 21:30:51.679848 8188 leader_election.cc:302] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 5099d51479c64cc9a3575f1ef3103e80; no voters:
I20260926 21:30:51.680152 8476 raft_consensus.cc:2781] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:51.680583 8476 raft_consensus.cc:686] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 1 LEADER]: Becoming Leader. State: Replica: 5099d51479c64cc9a3575f1ef3103e80, State: Running, Role: LEADER
I20260926 21:30:51.681043 8476 consensus_queue.cc:237] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
W20260926 21:30:51.683265 8308 tablet.cc:2369] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:51.685742 7960 catalog_manager.cc:5495] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } health_report { overall_health: UNKNOWN } } }
W20260926 21:30:51.689898 8447 tablet.cc:2369] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:30:51.736976 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:51.731755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.97.2:43587)
0926 21:30:51.731989 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:51.731995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:51.732013 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:51.732238 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:51.732242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:51.732316 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:51.732438 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:51.732775 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.733503 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.735074 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:51.735460 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:51.735551 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:51.735920 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:51.736426 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:51.736530 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:51.736626 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 21:30:51.736685 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 21:30:51.744871 8398 raft_consensus.cc:1261] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Refusing update from remote peer 5099d51479c64cc9a3575f1ef3103e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:51.744871 8123 raft_consensus.cc:1261] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Refusing update from remote peer 5099d51479c64cc9a3575f1ef3103e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:51.745796 8481 consensus_queue.cc:1035] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:51.746132 8476 consensus_queue.cc:1035] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:51.757122 8484 mvcc.cc:205] Tried to move back new op lower bound from 7333716999137882112 to 7333716998889447424. Current Snapshot: MvccSnapshot[applied={T|T < 7333716999137882112}]
I20260926 21:30:51.770679 7960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:51.770963 7960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:51.774610 7960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: Sending DeleteTablet for 3 replicas of tablet c151fa447a0e4932a02b27e75c84d23a
I20260926 21:30:51.775790 8098 tablet_service.cc:1514] Processing DeleteTablet for tablet c151fa447a0e4932a02b27e75c84d23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:51 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42648
I20260926 21:30:51.775790 8237 tablet_service.cc:1514] Processing DeleteTablet for tablet c151fa447a0e4932a02b27e75c84d23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:51 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35102
I20260926 21:30:51.776829 8376 tablet_service.cc:1514] Processing DeleteTablet for tablet c151fa447a0e4932a02b27e75c84d23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:51 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59832
I20260926 21:30:51.780282 7960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:51.781742 7960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:51.786667 8496 tablet_replica.cc:330] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: stopping tablet replica
I20260926 21:30:51.788021 8496 raft_consensus.cc:2229] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:51.789263 8496 raft_consensus.cc:2258] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:51.789323 8495 tablet_replica.cc:330] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: stopping tablet replica
I20260926 21:30:51.789860 8495 raft_consensus.cc:2229] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:51.790611 8495 raft_consensus.cc:2258] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:51.791527 8376 tablet_service.cc:1467] Processing CreateTablet for tablet 87fe1d8879d5410baf071d277a0dd9fc (DEFAULT_TABLE table=test-table [id=7d14748238094597a57a702c52906499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:51.791674 8097 tablet_service.cc:1467] Processing CreateTablet for tablet 87fe1d8879d5410baf071d277a0dd9fc (DEFAULT_TABLE table=test-table [id=7d14748238094597a57a702c52906499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:51.792048 8376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fe1d8879d5410baf071d277a0dd9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:51.792232 8097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fe1d8879d5410baf071d277a0dd9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:51.793371 8496 ts_tablet_manager.cc:1905] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:51.795499 8237 tablet_service.cc:1467] Processing CreateTablet for tablet 87fe1d8879d5410baf071d277a0dd9fc (DEFAULT_TABLE table=test-table [id=7d14748238094597a57a702c52906499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:51.795902 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fe1d8879d5410baf071d277a0dd9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:51.796123 8495 ts_tablet_manager.cc:1905] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:51.798540 8473 tablet_bootstrap.cc:492] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: Bootstrap starting.
I20260926 21:30:51.808406 8470 tablet_bootstrap.cc:492] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: Bootstrap starting.
I20260926 21:30:51.808828 8472 tablet_bootstrap.cc:492] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: Bootstrap starting.
I20260926 21:30:51.810451 8472 tablet_bootstrap.cc:654] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:51.810761 8470 tablet_bootstrap.cc:654] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:51.813932 8473 tablet_bootstrap.cc:654] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:51.816045 8494 tablet_replica.cc:330] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: stopping tablet replica
I20260926 21:30:51.817854 8494 raft_consensus.cc:2229] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:51.818048 8495 ts_tablet_manager.cc:1918] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:51.818357 8495 log.cc:1196] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/c151fa447a0e4932a02b27e75c84d23a
I20260926 21:30:51.818909 8494 raft_consensus.cc:2258] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:51.819308 8495 ts_tablet_manager.cc:1939] T c151fa447a0e4932a02b27e75c84d23a P 5099d51479c64cc9a3575f1ef3103e80: Deleting consensus metadata
I20260926 21:30:51.820393 8473 tablet_bootstrap.cc:492] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: No bootstrap required, opened a new log
I20260926 21:30:51.820648 8473 ts_tablet_manager.cc:1397] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.003s
I20260926 21:30:51.821785 8473 raft_consensus.cc:348] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.822180 8473 raft_consensus.cc:374] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:51.822343 8473 raft_consensus.cc:729] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53621701c7d544b18cc974524b4362d7, State: Initialized, Role: FOLLOWER
I20260926 21:30:51.822834 8473 consensus_queue.cc:260] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.824847 8473 ts_tablet_manager.cc:1428] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 21:30:51.825178 8494 ts_tablet_manager.cc:1905] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:51.828019 8472 tablet_bootstrap.cc:492] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: No bootstrap required, opened a new log
I20260926 21:30:51.828202 8472 ts_tablet_manager.cc:1397] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.008s
I20260926 21:30:51.828919 8472 raft_consensus.cc:348] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.829245 8472 raft_consensus.cc:374] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:51.829387 8472 raft_consensus.cc:729] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49574611798d4fc2a9b902ea838e9ce7, State: Initialized, Role: FOLLOWER
I20260926 21:30:51.829723 8472 consensus_queue.cc:260] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.834167 7946 catalog_manager.cc:4841] TS 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587): tablet c151fa447a0e4932a02b27e75c84d23a (table test-table [id=b2d4078300f74d85b8dc44f7be6e8320]) successfully deleted
I20260926 21:30:51.834573 8472 ts_tablet_manager.cc:1428] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260926 21:30:51.835101 8470 tablet_bootstrap.cc:492] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: No bootstrap required, opened a new log
I20260926 21:30:51.835326 8470 ts_tablet_manager.cc:1397] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.011s
I20260926 21:30:51.835989 8496 ts_tablet_manager.cc:1918] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:51.836233 8496 log.cc:1196] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/c151fa447a0e4932a02b27e75c84d23a
I20260926 21:30:51.836233 8470 raft_consensus.cc:348] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.836527 8470 raft_consensus.cc:374] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:51.836678 8470 raft_consensus.cc:729] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5099d51479c64cc9a3575f1ef3103e80, State: Initialized, Role: FOLLOWER
I20260926 21:30:51.836840 8496 ts_tablet_manager.cc:1939] T c151fa447a0e4932a02b27e75c84d23a P 53621701c7d544b18cc974524b4362d7: Deleting consensus metadata
I20260926 21:30:51.837036 8470 consensus_queue.cc:260] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:51.838737 7948 catalog_manager.cc:4841] TS 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561): tablet c151fa447a0e4932a02b27e75c84d23a (table test-table [id=b2d4078300f74d85b8dc44f7be6e8320]) successfully deleted
I20260926 21:30:51.839632 8470 ts_tablet_manager.cc:1428] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 21:30:51.844239 8494 ts_tablet_manager.cc:1918] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:51.844413 8494 log.cc:1196] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/c151fa447a0e4932a02b27e75c84d23a
I20260926 21:30:51.844942 8494 ts_tablet_manager.cc:1939] T c151fa447a0e4932a02b27e75c84d23a P 49574611798d4fc2a9b902ea838e9ce7: Deleting consensus metadata
I20260926 21:30:51.846798 7945 catalog_manager.cc:4841] TS 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861): tablet c151fa447a0e4932a02b27e75c84d23a (table test-table [id=b2d4078300f74d85b8dc44f7be6e8320]) successfully deleted
I20260926 21:30:52.132809 8478 raft_consensus.cc:482] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:52.133068 8478 raft_consensus.cc:504] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.134545 8478 leader_election.cc:288] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861), 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587)
I20260926 21:30:52.142871 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.135316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58365 (local address 127.0.97.1:44861)
0926 21:30:52.135514 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.135522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.136592 (+ 1070us) server_negotiation.cc:407] Connection header received
0926 21:30:52.136789 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.136795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.136883 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.137001 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:52.137622 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.138880 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.141121 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.141909 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.142058 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.142158 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.142478 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.142602 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:30:52.142978 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.134889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44861 (local address 127.0.97.3:58365)
0926 21:30:52.136465 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:52.136486 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:52.136638 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:52.137020 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:52.137024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:52.137050 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:52.137402 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.137437 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.138999 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.139004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.140984 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.140993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.142076 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.142082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.142271 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.142296 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 21:30:52.142404 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 21:30:52.142581 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1457,"mutex_wait_us":7,"thread_start_us":908,"threads_started":1}
I20260926 21:30:52.143803 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.134969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52271 (local address 127.0.97.2:43587)
0926 21:30:52.135221 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.135230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.136258 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 21:30:52.136539 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.136543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.136657 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.136764 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:52.137520 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.138465 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.140924 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.142461 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.142634 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.142732 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.143371 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.143509 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 21:30:52.143786 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fe1d8879d5410baf071d277a0dd9fc" candidate_uuid: "53621701c7d544b18cc974524b4362d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49574611798d4fc2a9b902ea838e9ce7" is_pre_election: true
I20260926 21:30:52.143893 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.134837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43587 (local address 127.0.97.3:52271)
0926 21:30:52.136110 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:52.136150 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:52.136376 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:52.136871 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:52.136880 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:52.136916 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:52.137285 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.137310 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.138587 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.138594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.140730 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.140744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.142925 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.142936 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.143190 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.143228 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 21:30:52.143345 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 21:30:52.143487 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":35,"thread_start_us":1316,"threads_started":1}
I20260926 21:30:52.144068 8123 raft_consensus.cc:2454] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53621701c7d544b18cc974524b4362d7 in term 0.
I20260926 21:30:52.144603 8327 leader_election.cc:302] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 53621701c7d544b18cc974524b4362d7; no voters:
I20260926 21:30:52.145038 8478 raft_consensus.cc:2781] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:52.145160 8478 raft_consensus.cc:482] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:52.145256 8478 raft_consensus.cc:3037] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.145677 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fe1d8879d5410baf071d277a0dd9fc" candidate_uuid: "53621701c7d544b18cc974524b4362d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5099d51479c64cc9a3575f1ef3103e80" is_pre_election: true
I20260926 21:30:52.146090 8262 raft_consensus.cc:2454] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53621701c7d544b18cc974524b4362d7 in term 0.
I20260926 21:30:52.147655 8478 raft_consensus.cc:504] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.148274 8478 leader_election.cc:288] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [CANDIDATE]: Term 1 election: Requested vote from peers 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861), 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587)
I20260926 21:30:52.148772 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fe1d8879d5410baf071d277a0dd9fc" candidate_uuid: "53621701c7d544b18cc974524b4362d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49574611798d4fc2a9b902ea838e9ce7"
I20260926 21:30:52.148860 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fe1d8879d5410baf071d277a0dd9fc" candidate_uuid: "53621701c7d544b18cc974524b4362d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5099d51479c64cc9a3575f1ef3103e80"
I20260926 21:30:52.149003 8123 raft_consensus.cc:3037] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.149055 8262 raft_consensus.cc:3037] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.152175 8123 raft_consensus.cc:2454] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53621701c7d544b18cc974524b4362d7 in term 1.
I20260926 21:30:52.152320 8262 raft_consensus.cc:2454] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53621701c7d544b18cc974524b4362d7 in term 1.
I20260926 21:30:52.152674 8327 leader_election.cc:302] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 53621701c7d544b18cc974524b4362d7; no voters:
I20260926 21:30:52.153044 8478 raft_consensus.cc:2781] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:52.153499 8478 raft_consensus.cc:686] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 1 LEADER]: Becoming Leader. State: Replica: 53621701c7d544b18cc974524b4362d7, State: Running, Role: LEADER
I20260926 21:30:52.153877 8478 consensus_queue.cc:237] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.157549 7961 catalog_manager.cc:5495] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53621701c7d544b18cc974524b4362d7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "53621701c7d544b18cc974524b4362d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } health_report { overall_health: HEALTHY } } }
Sep 26 21:30:52 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:52.226986 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.213011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.0.97.62:37005)
0926 21:30:52.214076 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.214082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.214103 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:30:52.214162 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.214164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.214211 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.214323 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:52.215491 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.216308 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.217547 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.217965 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.218095 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.221530 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:52.221535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:52.221539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:52.221565 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:52.224248 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:52.224949 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:52.224957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:52.224961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:52.225082 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:52.225437 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:52.225445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:52.225448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:52.225704 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:52.225854 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.226296 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.226558 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":943,"thread_start_us":681,"threads_started":1}
I20260926 21:30:52.227063 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.212986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37005 (local address 127.0.0.1:47386)
0926 21:30:52.213784 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:52.213806 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:52.213981 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:52.214351 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:52.214357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:52.215093 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:52.215322 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.215331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.216444 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.216451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.217394 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.217406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.218138 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.218143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.218325 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.219125 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:52.219144 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:52.221297 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:52.224397 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:52.224409 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:52.224417 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:52.224827 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:52.225193 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:52.225201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:52.225204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:52.225335 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:52.225830 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:52.225838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:52.226019 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 21:30:52.226218 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 21:30:52.226679 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":185,"thread_start_us":457,"threads_started":1}
I20260926 21:30:52.233131 7961 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47386:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 21:30:52.234581 7961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:52.241995 8097 tablet_service.cc:1467] Processing CreateTablet for tablet d41ba22eca174961846637aa6ea8c1ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=439d083f455f4785963f18c3b0b8e8e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:52.242599 8097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41ba22eca174961846637aa6ea8c1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.243279 8376 tablet_service.cc:1467] Processing CreateTablet for tablet d41ba22eca174961846637aa6ea8c1ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=439d083f455f4785963f18c3b0b8e8e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:52.243753 8376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41ba22eca174961846637aa6ea8c1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.249287 8473 tablet_bootstrap.cc:492] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: Bootstrap starting.
I20260926 21:30:52.250496 8472 tablet_bootstrap.cc:492] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: Bootstrap starting.
I20260926 21:30:52.250862 8237 tablet_service.cc:1467] Processing CreateTablet for tablet d41ba22eca174961846637aa6ea8c1ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=439d083f455f4785963f18c3b0b8e8e4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:30:52.251071 8473 tablet_bootstrap.cc:654] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.251328 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d41ba22eca174961846637aa6ea8c1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.252223 8472 tablet_bootstrap.cc:654] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.253562 8473 tablet_bootstrap.cc:492] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: No bootstrap required, opened a new log
I20260926 21:30:52.253747 8473 ts_tablet_manager.cc:1397] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 21:30:52.254231 8472 tablet_bootstrap.cc:492] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: No bootstrap required, opened a new log
I20260926 21:30:52.254405 8472 ts_tablet_manager.cc:1397] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 21:30:52.254581 8473 raft_consensus.cc:348] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.254891 8473 raft_consensus.cc:374] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.254987 8473 raft_consensus.cc:729] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53621701c7d544b18cc974524b4362d7, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.255121 8472 raft_consensus.cc:348] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.255367 8472 raft_consensus.cc:374] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.255272 8473 consensus_queue.cc:260] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.255445 8472 raft_consensus.cc:729] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49574611798d4fc2a9b902ea838e9ce7, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.255712 8472 consensus_queue.cc:260] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.255863 8473 ts_tablet_manager.cc:1428] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:52.255748 8478 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.256024 8478 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:52.256156 8477 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.256464 8477 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:52.257689 8472 ts_tablet_manager.cc:1428] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260926 21:30:52.259393 8470 tablet_bootstrap.cc:492] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: Bootstrap starting.
I20260926 21:30:52.260944 8470 tablet_bootstrap.cc:654] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.262836 8470 tablet_bootstrap.cc:492] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: No bootstrap required, opened a new log
I20260926 21:30:52.262996 8470 ts_tablet_manager.cc:1397] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260926 21:30:52.263567 8470 raft_consensus.cc:348] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.263772 8470 raft_consensus.cc:374] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.263872 8470 raft_consensus.cc:729] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5099d51479c64cc9a3575f1ef3103e80, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.264120 8470 consensus_queue.cc:260] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.264549 8476 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.264678 8470 ts_tablet_manager.cc:1428] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 21:30:52.264823 8476 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:52.327521 8476 raft_consensus.cc:482] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:52.327692 8476 raft_consensus.cc:504] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.328401 8476 leader_election.cc:288] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861), 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:52.328884 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41ba22eca174961846637aa6ea8c1ed" candidate_uuid: "5099d51479c64cc9a3575f1ef3103e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49574611798d4fc2a9b902ea838e9ce7" is_pre_election: true
I20260926 21:30:52.328929 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41ba22eca174961846637aa6ea8c1ed" candidate_uuid: "5099d51479c64cc9a3575f1ef3103e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53621701c7d544b18cc974524b4362d7" is_pre_election: true
I20260926 21:30:52.329120 8123 raft_consensus.cc:2454] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5099d51479c64cc9a3575f1ef3103e80 in term 0.
I20260926 21:30:52.329226 8398 raft_consensus.cc:2454] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5099d51479c64cc9a3575f1ef3103e80 in term 0.
I20260926 21:30:52.329545 8188 leader_election.cc:302] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 5099d51479c64cc9a3575f1ef3103e80; no voters:
I20260926 21:30:52.329797 8476 raft_consensus.cc:2781] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:52.329918 8476 raft_consensus.cc:482] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:52.330006 8476 raft_consensus.cc:3037] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.332218 8476 raft_consensus.cc:504] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.332811 8476 leader_election.cc:288] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [CANDIDATE]: Term 1 election: Requested vote from peers 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861), 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:52.333295 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41ba22eca174961846637aa6ea8c1ed" candidate_uuid: "5099d51479c64cc9a3575f1ef3103e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49574611798d4fc2a9b902ea838e9ce7"
I20260926 21:30:52.333343 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d41ba22eca174961846637aa6ea8c1ed" candidate_uuid: "5099d51479c64cc9a3575f1ef3103e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53621701c7d544b18cc974524b4362d7"
I20260926 21:30:52.333500 8123 raft_consensus.cc:3037] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.333596 8398 raft_consensus.cc:3037] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.335407 8123 raft_consensus.cc:2454] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5099d51479c64cc9a3575f1ef3103e80 in term 1.
I20260926 21:30:52.335806 8188 leader_election.cc:302] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 5099d51479c64cc9a3575f1ef3103e80; no voters:
I20260926 21:30:52.336000 8398 raft_consensus.cc:2454] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5099d51479c64cc9a3575f1ef3103e80 in term 1.
I20260926 21:30:52.336094 8476 raft_consensus.cc:2781] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:52.336476 8476 raft_consensus.cc:686] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [term 1 LEADER]: Becoming Leader. State: Replica: 5099d51479c64cc9a3575f1ef3103e80, State: Running, Role: LEADER
I20260926 21:30:52.336789 8476 consensus_queue.cc:237] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.338294 8514 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: TxnStatusTablet state changed. Reason: New leader 5099d51479c64cc9a3575f1ef3103e80. Latest consensus state: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.338819 8514 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:52.339398 8516 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 21:30:52.339624 8516 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 21:30:52.339978 7961 catalog_manager.cc:5495] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } health_report { overall_health: UNKNOWN } } }
I20260926 21:30:52.356317 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.350957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43587 (local address 127.0.0.1:35130)
0926 21:30:52.351195 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:52.351212 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:52.351322 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:52.351704 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:52.351707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:52.351717 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:52.351853 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.351859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.353305 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.353308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.354275 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.354286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.355307 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.355311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.355443 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.355502 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:52.355913 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:52.355928 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 21:30:52.355989 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 21:30:52.356100 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:30:52.356420 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.351021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.0.97.2:43587)
0926 21:30:52.351354 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.351360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.351384 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:30:52.351477 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.351482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.351564 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.351661 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:52.351978 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.353161 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.354450 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.355147 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.355269 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.355689 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:52.355816 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:52.355909 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.356038 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.356126 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260926 21:30:52.360589 8398 raft_consensus.cc:1261] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Refusing update from remote peer 5099d51479c64cc9a3575f1ef3103e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:52.360625 8123 raft_consensus.cc:1261] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Refusing update from remote peer 5099d51479c64cc9a3575f1ef3103e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:52.361271 8476 consensus_queue.cc:1035] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:52.361543 8514 consensus_queue.cc:1035] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:52.369293 8477 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: TxnStatusTablet state changed. Reason: New leader 5099d51479c64cc9a3575f1ef3103e80. Latest consensus state: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.369650 8477 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:52.372567 8514 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.372872 8514 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:52.376539 8478 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: TxnStatusTablet state changed. Reason: New leader 5099d51479c64cc9a3575f1ef3103e80. Latest consensus state: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.376817 8478 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:52.378232 8476 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.378623 8476 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 5099d51479c64cc9a3575f1ef3103e80: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:30:52.384375 8477 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.384635 8477 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 49574611798d4fc2a9b902ea838e9ce7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:52.385913 8478 tablet_replica.cc:439] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5099d51479c64cc9a3575f1ef3103e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } } }
I20260926 21:30:52.386207 8478 tablet_replica.cc:442] T d41ba22eca174961846637aa6ea8c1ed P 53621701c7d544b18cc974524b4362d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:30:52.402544 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.385529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.0.97.3:46561)
0926 21:30:52.394069 (+ 8540us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.394076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.394951 (+ 875us) server_negotiation.cc:407] Connection header received
0926 21:30:52.395226 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.395230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.395318 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.395458 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:52.395962 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.397113 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.399207 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.399983 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.400177 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.400618 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:52.401408 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:52.401556 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.402062 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.402147 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8402,"thread_start_us":8247,"threads_started":1}
I20260926 21:30:52.423843 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.417574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43587 (local address 127.0.97.3:40855)
0926 21:30:52.418235 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:52.418255 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:52.418411 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:52.418821 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:52.418826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:52.418838 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:52.419190 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.419200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.420637 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.420641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.421710 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.421721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.422768 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.422772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.422886 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.422940 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:52.423364 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:52.423378 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 21:30:52.423436 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 21:30:52.423537 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":315,"threads_started":1}
I20260926 21:30:52.423875 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.417560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40855 (local address 127.0.97.2:43587)
0926 21:30:52.417726 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.417733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.418364 (+ 631us) server_negotiation.cc:407] Connection header received
0926 21:30:52.418592 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.418598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.418686 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.418798 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:52.419340 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.420496 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.421851 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.422599 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.422734 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.423110 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:52.423270 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:52.423359 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.423490 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.423576 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:30:52.435002 8123 raft_consensus.cc:1261] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Refusing update from remote peer 53621701c7d544b18cc974524b4362d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:52.435724 8502 consensus_queue.cc:1035] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:52.436031 8262 raft_consensus.cc:1261] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Refusing update from remote peer 53621701c7d544b18cc974524b4362d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:52.436702 8502 consensus_queue.cc:1035] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:52.459012 8519 mvcc.cc:205] Tried to move back new op lower bound from 7333717001968451584 to 7333717000826626048. Current Snapshot: MvccSnapshot[applied={T|T < 7333717001968451584}]
I20260926 21:30:52.495309 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.488185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48129 (local address 127.0.97.3:46561)
0926 21:30:52.488346 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.488352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.489056 (+ 704us) server_negotiation.cc:407] Connection header received
0926 21:30:52.489292 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.489300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.489386 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.489482 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:52.490020 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.491171 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.493456 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.493904 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.494012 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.494640 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:52.494756 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:52.494827 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.495020 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.495087 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 21:30:52.495841 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.488103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46561 (local address 127.0.97.2:48129)
0926 21:30:52.488925 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:52.488941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:52.489124 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:52.489503 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:52.489507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:52.489516 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:52.489834 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.489846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.491346 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.491352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.493273 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.493291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.494117 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.494124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.494306 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.494386 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:52.494865 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:52.494877 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 21:30:52.495008 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 21:30:52.495165 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":450,"threads_started":1}
I20260926 21:30:52.541448 7960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:52.541733 7960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:52.545732 7960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: Sending DeleteTablet for 3 replicas of tablet 87fe1d8879d5410baf071d277a0dd9fc
I20260926 21:30:52.548810 8097 tablet_service.cc:1514] Processing DeleteTablet for tablet 87fe1d8879d5410baf071d277a0dd9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42648
I20260926 21:30:52.549494 8376 tablet_service.cc:1514] Processing DeleteTablet for tablet 87fe1d8879d5410baf071d277a0dd9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59832
I20260926 21:30:52.553959 8237 tablet_service.cc:1514] Processing DeleteTablet for tablet 87fe1d8879d5410baf071d277a0dd9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35102
I20260926 21:30:52.562913 8540 tablet_replica.cc:330] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: stopping tablet replica
I20260926 21:30:52.563212 8540 raft_consensus.cc:2229] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:52.564010 8540 raft_consensus.cc:2258] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:52.565402 8540 ts_tablet_manager.cc:1905] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:52.566197 8541 tablet_replica.cc:330] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: stopping tablet replica
I20260926 21:30:52.566623 8541 raft_consensus.cc:2229] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:52.567006 8541 raft_consensus.cc:2258] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:52.569432 8541 ts_tablet_manager.cc:1905] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:52.572645 8540 ts_tablet_manager.cc:1918] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:52.572788 8540 log.cc:1196] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/87fe1d8879d5410baf071d277a0dd9fc
I20260926 21:30:52.573246 8540 ts_tablet_manager.cc:1939] T 87fe1d8879d5410baf071d277a0dd9fc P 53621701c7d544b18cc974524b4362d7: Deleting consensus metadata
I20260926 21:30:52.574473 7948 catalog_manager.cc:4841] TS 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561): tablet 87fe1d8879d5410baf071d277a0dd9fc (table test-table [id=7d14748238094597a57a702c52906499]) successfully deleted
I20260926 21:30:52.576689 8539 tablet_replica.cc:330] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: stopping tablet replica
I20260926 21:30:52.576912 8539 raft_consensus.cc:2229] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:52.577204 8539 raft_consensus.cc:2258] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:52.577418 8541 ts_tablet_manager.cc:1918] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:52.577554 8541 log.cc:1196] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/87fe1d8879d5410baf071d277a0dd9fc
I20260926 21:30:52.578003 8541 ts_tablet_manager.cc:1939] T 87fe1d8879d5410baf071d277a0dd9fc P 5099d51479c64cc9a3575f1ef3103e80: Deleting consensus metadata
I20260926 21:30:52.579248 8539 ts_tablet_manager.cc:1905] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:52.579586 7946 catalog_manager.cc:4841] TS 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587): tablet 87fe1d8879d5410baf071d277a0dd9fc (table test-table [id=7d14748238094597a57a702c52906499]) successfully deleted
I20260926 21:30:52.587452 8539 ts_tablet_manager.cc:1918] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:52.588461 8539 log.cc:1196] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/87fe1d8879d5410baf071d277a0dd9fc
I20260926 21:30:52.589097 8539 ts_tablet_manager.cc:1939] T 87fe1d8879d5410baf071d277a0dd9fc P 49574611798d4fc2a9b902ea838e9ce7: Deleting consensus metadata
I20260926 21:30:52.590569 7945 catalog_manager.cc:4841] TS 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861): tablet 87fe1d8879d5410baf071d277a0dd9fc (table test-table [id=7d14748238094597a57a702c52906499]) successfully deleted
Sep 26 21:30:52 dist-test-slave-87bt krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:30:52 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:30:52 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 21:30:52.650291 8510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:30:52.636654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.0.97.62:37005)
0926 21:30:52.636847 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.636853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.637397 (+ 544us) server_negotiation.cc:407] Connection header received
0926 21:30:52.637582 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.637589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.637694 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.637822 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:52.639552 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.640952 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.642797 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.643532 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.643676 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.649996 (+ 6320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47394: BlockingRecv error: recv got EOF from 127.0.0.1:47394 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 21:30:52 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:30:52.674578 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.654810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.0.97.62:37005)
0926 21:30:52.654982 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.654988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.655592 (+ 604us) server_negotiation.cc:407] Connection header received
0926 21:30:52.655879 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.655887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.655997 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.656120 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:52.657608 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.658931 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.660816 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.661573 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.661715 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.667127 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:52.667135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:52.667140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:52.667206 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:52.671190 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:52.672157 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:52.672164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:52.672168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:52.672331 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:52.672821 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:52.672826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:52.672832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:52.673230 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:52.673517 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.673891 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.674162 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:30:52.679327 7960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:52.680044 7960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:52.687832 8097 tablet_service.cc:1467] Processing CreateTablet for tablet de4ad2faf87541cea08da33d0eeebbff (DEFAULT_TABLE table=test-table [id=538bfcc4739e42b9b9a5c0ea0708631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:52.688330 8097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4ad2faf87541cea08da33d0eeebbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.689389 8376 tablet_service.cc:1467] Processing CreateTablet for tablet de4ad2faf87541cea08da33d0eeebbff (DEFAULT_TABLE table=test-table [id=538bfcc4739e42b9b9a5c0ea0708631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:52.689878 8376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4ad2faf87541cea08da33d0eeebbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.690913 8237 tablet_service.cc:1467] Processing CreateTablet for tablet de4ad2faf87541cea08da33d0eeebbff (DEFAULT_TABLE table=test-table [id=538bfcc4739e42b9b9a5c0ea0708631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:52.691375 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4ad2faf87541cea08da33d0eeebbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.695032 8472 tablet_bootstrap.cc:492] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: Bootstrap starting.
I20260926 21:30:52.696693 8472 tablet_bootstrap.cc:654] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.698935 8470 tablet_bootstrap.cc:492] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: Bootstrap starting.
I20260926 21:30:52.700946 8470 tablet_bootstrap.cc:654] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.703576 8473 tablet_bootstrap.cc:492] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: Bootstrap starting.
I20260926 21:30:52.705443 8473 tablet_bootstrap.cc:654] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.709319 8472 tablet_bootstrap.cc:492] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: No bootstrap required, opened a new log
I20260926 21:30:52.709489 8472 ts_tablet_manager.cc:1397] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260926 21:30:52.709954 8473 tablet_bootstrap.cc:492] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: No bootstrap required, opened a new log
I20260926 21:30:52.710171 8473 ts_tablet_manager.cc:1397] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260926 21:30:52.710333 8472 raft_consensus.cc:348] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.710631 8472 raft_consensus.cc:374] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.710733 8472 raft_consensus.cc:729] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49574611798d4fc2a9b902ea838e9ce7, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.711010 8472 consensus_queue.cc:260] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.711185 8473 raft_consensus.cc:348] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.711510 8473 raft_consensus.cc:374] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.711653 8472 ts_tablet_manager.cc:1428] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 21:30:52.711668 8473 raft_consensus.cc:729] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53621701c7d544b18cc974524b4362d7, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.711941 8473 consensus_queue.cc:260] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.712604 8473 ts_tablet_manager.cc:1428] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:52.716252 8470 tablet_bootstrap.cc:492] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: No bootstrap required, opened a new log
I20260926 21:30:52.718020 8470 ts_tablet_manager.cc:1397] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20260926 21:30:52.718832 8470 raft_consensus.cc:348] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.719056 8470 raft_consensus.cc:374] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.719156 8470 raft_consensus.cc:729] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5099d51479c64cc9a3575f1ef3103e80, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.719398 8470 consensus_queue.cc:260] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.721136 8470 ts_tablet_manager.cc:1428] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:52.731158 8477 raft_consensus.cc:482] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:52.731417 8477 raft_consensus.cc:504] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.733225 8477 leader_election.cc:288] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587), 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:52.754014 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.734104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57351 (local address 127.0.97.3:46561)
0926 21:30:52.734409 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.734453 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.746760 (+ 12307us) server_negotiation.cc:407] Connection header received
0926 21:30:52.746956 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.746963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.747053 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.747156 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:52.747902 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.749046 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.751475 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.752777 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.752890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.753017 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.753538 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.753678 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 21:30:52.754963 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.733682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46561 (local address 127.0.97.1:57351)
0926 21:30:52.746639 (+ 12957us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:52.746656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:52.746800 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:52.747345 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:52.747350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:52.747373 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:52.747703 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.747715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.749227 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.749235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.751276 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.751290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.753153 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.753160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.753337 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.753373 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 21:30:52.754475 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0926 21:30:52.754612 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12839,"thread_start_us":10851,"threads_started":1}
I20260926 21:30:52.755173 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.734734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46743 (local address 127.0.97.2:43587)
0926 21:30:52.734970 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.734976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.744507 (+ 9531us) server_negotiation.cc:407] Connection header received
0926 21:30:52.745073 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.745082 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.745191 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.745277 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:30:52.747000 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.748209 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.752091 (+ 3882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.753452 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.753588 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.753707 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.754743 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.754879 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 21:30:52.756007 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.733347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43587 (local address 127.0.97.1:46743)
0926 21:30:52.744345 (+ 10998us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:52.744380 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:52.744856 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:52.745581 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:52.745589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:52.745631 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:30:52.746033 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.746049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.748394 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.748402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.751895 (+ 3493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:52.751909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.754354 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.754359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:52.754560 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.754591 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:30:52.755541 (+ 950us) client_negotiation.cc:240] Negotiation successful
0926 21:30:52.755681 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10805,"thread_start_us":13015,"threads_started":1}
I20260926 21:30:52.755928 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ad2faf87541cea08da33d0eeebbff" candidate_uuid: "49574611798d4fc2a9b902ea838e9ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53621701c7d544b18cc974524b4362d7" is_pre_election: true
I20260926 21:30:52.756220 8398 raft_consensus.cc:2454] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49574611798d4fc2a9b902ea838e9ce7 in term 0.
I20260926 21:30:52.756754 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ad2faf87541cea08da33d0eeebbff" candidate_uuid: "49574611798d4fc2a9b902ea838e9ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5099d51479c64cc9a3575f1ef3103e80" is_pre_election: true
I20260926 21:30:52.756835 8051 leader_election.cc:302] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 53621701c7d544b18cc974524b4362d7; no voters:
I20260926 21:30:52.757035 8262 raft_consensus.cc:2454] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49574611798d4fc2a9b902ea838e9ce7 in term 0.
I20260926 21:30:52.757308 8477 raft_consensus.cc:2781] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:52.757431 8477 raft_consensus.cc:482] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:52.757539 8477 raft_consensus.cc:3037] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.759814 8477 raft_consensus.cc:504] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.760846 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ad2faf87541cea08da33d0eeebbff" candidate_uuid: "49574611798d4fc2a9b902ea838e9ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5099d51479c64cc9a3575f1ef3103e80"
I20260926 21:30:52.761049 8477 leader_election.cc:288] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [CANDIDATE]: Term 1 election: Requested vote from peers 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587), 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:52.761080 8262 raft_consensus.cc:3037] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.761871 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ad2faf87541cea08da33d0eeebbff" candidate_uuid: "49574611798d4fc2a9b902ea838e9ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53621701c7d544b18cc974524b4362d7"
I20260926 21:30:52.762087 8398 raft_consensus.cc:3037] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:52.764151 8398 raft_consensus.cc:2454] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49574611798d4fc2a9b902ea838e9ce7 in term 1.
I20260926 21:30:52.764571 8051 leader_election.cc:302] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 53621701c7d544b18cc974524b4362d7; no voters:
I20260926 21:30:52.764529 8262 raft_consensus.cc:2454] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49574611798d4fc2a9b902ea838e9ce7 in term 1.
I20260926 21:30:52.764864 8477 raft_consensus.cc:2781] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:52.774086 8477 raft_consensus.cc:686] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 49574611798d4fc2a9b902ea838e9ce7, State: Running, Role: LEADER
I20260926 21:30:52.774582 8477 consensus_queue.cc:237] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.778612 7961 catalog_manager.cc:5495] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "49574611798d4fc2a9b902ea838e9ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } health_report { overall_health: UNKNOWN } } }
I20260926 21:30:52.826344 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:52.808239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.97.1:44861)
0926 21:30:52.818945 (+ 10706us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:52.818952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:52.819002 (+ 50us) server_negotiation.cc:407] Connection header received
0926 21:30:52.819136 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:52.819143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:52.819286 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:52.819535 (+ 249us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:52.820071 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.821229 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.823298 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:52.823964 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:52.824559 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:52.824719 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:52.825493 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:52.825666 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:52.825807 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 21:30:52.825887 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10533,"thread_start_us":10365,"threads_started":1}
I20260926 21:30:52.830363 8262 raft_consensus.cc:1261] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Refusing update from remote peer 49574611798d4fc2a9b902ea838e9ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:52.831708 8559 consensus_queue.cc:1035] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:52.836197 8398 raft_consensus.cc:1261] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Refusing update from remote peer 49574611798d4fc2a9b902ea838e9ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:52.837966 8559 consensus_queue.cc:1035] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:52.874580 7961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:52.874886 7961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:52.879122 7961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: Sending DeleteTablet for 3 replicas of tablet de4ad2faf87541cea08da33d0eeebbff
I20260926 21:30:52.880362 8097 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ad2faf87541cea08da33d0eeebbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42648
I20260926 21:30:52.880700 8539 tablet_replica.cc:330] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: stopping tablet replica
I20260926 21:30:52.880656 8237 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ad2faf87541cea08da33d0eeebbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35102
I20260926 21:30:52.880924 8539 raft_consensus.cc:2229] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:52.881059 8541 tablet_replica.cc:330] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: stopping tablet replica
I20260926 21:30:52.881074 8376 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ad2faf87541cea08da33d0eeebbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59832
I20260926 21:30:52.881242 8541 raft_consensus.cc:2229] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:52.881493 8541 raft_consensus.cc:2258] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:52.881496 8539 raft_consensus.cc:2258] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:52.881546 8540 tablet_replica.cc:330] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: stopping tablet replica
I20260926 21:30:52.881750 8540 raft_consensus.cc:2229] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:52.881992 8540 raft_consensus.cc:2258] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:52.882993 7961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:30:52.884266 7961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:30:52.888223 8540 ts_tablet_manager.cc:1905] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:52.889539 8539 ts_tablet_manager.cc:1905] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:52.890784 8541 ts_tablet_manager.cc:1905] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:52.893523 8237 tablet_service.cc:1467] Processing CreateTablet for tablet c67b84efbf0c4767ba539324e57a64cf (DEFAULT_TABLE table=test-table [id=7e4363baac514e94bb7b0252db0887e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:52.893980 8237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67b84efbf0c4767ba539324e57a64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.896615 8097 tablet_service.cc:1467] Processing CreateTablet for tablet c67b84efbf0c4767ba539324e57a64cf (DEFAULT_TABLE table=test-table [id=7e4363baac514e94bb7b0252db0887e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:52.897049 8097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67b84efbf0c4767ba539324e57a64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.901705 8470 tablet_bootstrap.cc:492] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: Bootstrap starting.
I20260926 21:30:52.903529 8470 tablet_bootstrap.cc:654] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.903884 8472 tablet_bootstrap.cc:492] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: Bootstrap starting.
I20260926 21:30:52.905881 8472 tablet_bootstrap.cc:654] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.906044 8541 ts_tablet_manager.cc:1918] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:52.906201 8541 log.cc:1196] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/de4ad2faf87541cea08da33d0eeebbff
I20260926 21:30:52.907727 8470 tablet_bootstrap.cc:492] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: No bootstrap required, opened a new log
I20260926 21:30:52.907953 8470 ts_tablet_manager.cc:1397] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 21:30:52.907888 8376 tablet_service.cc:1467] Processing CreateTablet for tablet c67b84efbf0c4767ba539324e57a64cf (DEFAULT_TABLE table=test-table [id=7e4363baac514e94bb7b0252db0887e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:30:52.908349 8376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67b84efbf0c4767ba539324e57a64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:52.908771 8470 raft_consensus.cc:348] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.909006 8470 raft_consensus.cc:374] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.909082 8470 raft_consensus.cc:729] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5099d51479c64cc9a3575f1ef3103e80, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.909358 8470 consensus_queue.cc:260] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.911214 8470 ts_tablet_manager.cc:1428] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 21:30:52.912074 8541 ts_tablet_manager.cc:1939] T de4ad2faf87541cea08da33d0eeebbff P 5099d51479c64cc9a3575f1ef3103e80: Deleting consensus metadata
I20260926 21:30:52.913766 7946 catalog_manager.cc:4841] TS 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587): tablet de4ad2faf87541cea08da33d0eeebbff (table test-table [id=538bfcc4739e42b9b9a5c0ea0708631b]) successfully deleted
I20260926 21:30:52.915148 8473 tablet_bootstrap.cc:492] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: Bootstrap starting.
I20260926 21:30:52.916800 8473 tablet_bootstrap.cc:654] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:52.920027 8540 ts_tablet_manager.cc:1918] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:52.920209 8540 log.cc:1196] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/de4ad2faf87541cea08da33d0eeebbff
I20260926 21:30:52.920708 8540 ts_tablet_manager.cc:1939] T de4ad2faf87541cea08da33d0eeebbff P 53621701c7d544b18cc974524b4362d7: Deleting consensus metadata
I20260926 21:30:52.923087 7948 catalog_manager.cc:4841] TS 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561): tablet de4ad2faf87541cea08da33d0eeebbff (table test-table [id=538bfcc4739e42b9b9a5c0ea0708631b]) successfully deleted
I20260926 21:30:52.924701 8472 tablet_bootstrap.cc:492] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: No bootstrap required, opened a new log
I20260926 21:30:52.924885 8472 ts_tablet_manager.cc:1397] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20260926 21:30:52.925719 8472 raft_consensus.cc:348] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.926136 8472 raft_consensus.cc:374] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.926270 8472 raft_consensus.cc:729] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49574611798d4fc2a9b902ea838e9ce7, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.926635 8472 consensus_queue.cc:260] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.928829 8473 tablet_bootstrap.cc:492] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: No bootstrap required, opened a new log
I20260926 21:30:52.929008 8473 ts_tablet_manager.cc:1397] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20260926 21:30:52.929654 8472 ts_tablet_manager.cc:1428] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 21:30:52.929862 8473 raft_consensus.cc:348] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.930166 8473 raft_consensus.cc:374] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:52.930253 8473 raft_consensus.cc:729] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53621701c7d544b18cc974524b4362d7, State: Initialized, Role: FOLLOWER
I20260926 21:30:52.930557 8473 consensus_queue.cc:260] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:52.931078 8473 ts_tablet_manager.cc:1428] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:30:52.937547 8539 ts_tablet_manager.cc:1918] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:30:52.937769 8539 log.cc:1196] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/de4ad2faf87541cea08da33d0eeebbff
I20260926 21:30:52.938263 8539 ts_tablet_manager.cc:1939] T de4ad2faf87541cea08da33d0eeebbff P 49574611798d4fc2a9b902ea838e9ce7: Deleting consensus metadata
I20260926 21:30:52.939489 7945 catalog_manager.cc:4841] TS 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861): tablet de4ad2faf87541cea08da33d0eeebbff (table test-table [id=538bfcc4739e42b9b9a5c0ea0708631b]) successfully deleted
I20260926 21:30:53.104231 8559 raft_consensus.cc:482] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:30:53.104481 8559 raft_consensus.cc:504] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:53.105289 8559 leader_election.cc:288] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587), 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:53.105860 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67b84efbf0c4767ba539324e57a64cf" candidate_uuid: "49574611798d4fc2a9b902ea838e9ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5099d51479c64cc9a3575f1ef3103e80" is_pre_election: true
I20260926 21:30:53.106045 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67b84efbf0c4767ba539324e57a64cf" candidate_uuid: "49574611798d4fc2a9b902ea838e9ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53621701c7d544b18cc974524b4362d7" is_pre_election: true
I20260926 21:30:53.106119 8262 raft_consensus.cc:2454] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49574611798d4fc2a9b902ea838e9ce7 in term 0.
I20260926 21:30:53.106307 8398 raft_consensus.cc:2454] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49574611798d4fc2a9b902ea838e9ce7 in term 0.
I20260926 21:30:53.106791 8049 leader_election.cc:302] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 5099d51479c64cc9a3575f1ef3103e80; no voters:
I20260926 21:30:53.107173 8559 raft_consensus.cc:2781] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:30:53.107297 8559 raft_consensus.cc:482] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:30:53.107384 8559 raft_consensus.cc:3037] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:53.110627 8559 raft_consensus.cc:504] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:53.111352 8559 leader_election.cc:288] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [CANDIDATE]: Term 1 election: Requested vote from peers 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587), 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561)
I20260926 21:30:53.111881 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67b84efbf0c4767ba539324e57a64cf" candidate_uuid: "49574611798d4fc2a9b902ea838e9ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5099d51479c64cc9a3575f1ef3103e80"
I20260926 21:30:53.112076 8262 raft_consensus.cc:3037] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:53.112027 8398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67b84efbf0c4767ba539324e57a64cf" candidate_uuid: "49574611798d4fc2a9b902ea838e9ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53621701c7d544b18cc974524b4362d7"
I20260926 21:30:53.112249 8398 raft_consensus.cc:3037] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:53.114967 8262 raft_consensus.cc:2454] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49574611798d4fc2a9b902ea838e9ce7 in term 1.
I20260926 21:30:53.115427 8049 leader_election.cc:302] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49574611798d4fc2a9b902ea838e9ce7, 5099d51479c64cc9a3575f1ef3103e80; no voters:
I20260926 21:30:53.115767 8559 raft_consensus.cc:2781] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:53.115998 8559 raft_consensus.cc:686] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 49574611798d4fc2a9b902ea838e9ce7, State: Running, Role: LEADER
I20260926 21:30:53.115895 8398 raft_consensus.cc:2454] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49574611798d4fc2a9b902ea838e9ce7 in term 1.
I20260926 21:30:53.116271 8559 consensus_queue.cc:237] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } }
I20260926 21:30:53.119155 7960 catalog_manager.cc:5495] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "49574611798d4fc2a9b902ea838e9ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49574611798d4fc2a9b902ea838e9ce7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 } health_report { overall_health: UNKNOWN } } }
I20260926 21:30:53.178174 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:53.170371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38395 (local address 127.0.97.2:43587)
0926 21:30:53.170583 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:53.170590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:53.171384 (+ 794us) server_negotiation.cc:407] Connection header received
0926 21:30:53.171698 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:53.171705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:53.171802 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:53.171864 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:53.172695 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.173760 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.175909 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.176454 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.176549 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:53.177333 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:53.177474 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:53.177534 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:53.177771 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 21:30:53.177857 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 21:30:53.179280 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:53.170464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43587 (local address 127.0.97.1:38395)
0926 21:30:53.171242 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:53.171268 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:53.171524 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:53.172071 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:53.172076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:53.172089 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:53.172489 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:53.172502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.173970 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.173980 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:53.175718 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:53.175735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.176818 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.176824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:53.177039 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:53.177112 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:53.177616 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:53.177626 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:30:53.178652 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0926 21:30:53.178831 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":376,"threads_started":1}
I20260926 21:30:53.189627 8398 raft_consensus.cc:1261] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Refusing update from remote peer 49574611798d4fc2a9b902ea838e9ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:53.189713 8262 raft_consensus.cc:1261] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Refusing update from remote peer 49574611798d4fc2a9b902ea838e9ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:30:53.190240 8477 consensus_queue.cc:1035] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5099d51479c64cc9a3575f1ef3103e80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:53.190627 8559 consensus_queue.cc:1035] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53621701c7d544b18cc974524b4362d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:30:53.226118 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:53.214764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44861 (local address 127.0.97.2:43677)
0926 21:30:53.219743 (+ 4979us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:53.219765 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:53.219900 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:53.220268 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:53.220273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:53.220284 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:30:53.220566 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:53.220580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.221977 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.221981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:53.223209 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:53.223221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.224468 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.224477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:53.224973 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:53.225028 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:30:53.225579 (+ 551us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:30:53.225589 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:30:53.225761 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 21:30:53.225869 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4817,"thread_start_us":4665,"threads_started":1}
I20260926 21:30:53.226231 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:53.215219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43677 (local address 127.0.97.1:44861)
0926 21:30:53.215378 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:53.215384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:53.219857 (+ 4473us) server_negotiation.cc:407] Connection header received
0926 21:30:53.220037 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:53.220043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:53.220139 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:53.220232 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:30:53.220712 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.221841 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.223356 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.224051 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.224200 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:53.225331 (+ 1131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:30:53.225462 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:30:53.225565 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:53.225785 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 21:30:53.225903 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:30:53.261854 7960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:53.262162 7960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:30:53.265429 7960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c77c5e52ef804fc18cad445e8b0ce4df: Sending DeleteTablet for 3 replicas of tablet c67b84efbf0c4767ba539324e57a64cf
I20260926 21:30:53.266187 8097 tablet_service.cc:1514] Processing DeleteTablet for tablet c67b84efbf0c4767ba539324e57a64cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42648
I20260926 21:30:53.266264 8237 tablet_service.cc:1514] Processing DeleteTablet for tablet c67b84efbf0c4767ba539324e57a64cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35102
I20260926 21:30:53.266470 8376 tablet_service.cc:1514] Processing DeleteTablet for tablet c67b84efbf0c4767ba539324e57a64cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:30:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59832
I20260926 21:30:53.266523 8539 tablet_replica.cc:330] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: stopping tablet replica
I20260926 21:30:53.266613 8541 tablet_replica.cc:330] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: stopping tablet replica
I20260926 21:30:53.266736 8540 tablet_replica.cc:330] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: stopping tablet replica
I20260926 21:30:53.266710 8539 raft_consensus.cc:2229] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:30:53.266785 8541 raft_consensus.cc:2229] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:53.266885 8540 raft_consensus.cc:2229] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:30:53.266979 8541 raft_consensus.cc:2258] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:53.267055 8539 raft_consensus.cc:2258] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:53.267081 8540 raft_consensus.cc:2258] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:30:53.268189 8541 ts_tablet_manager.cc:1905] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:53.268218 8540 ts_tablet_manager.cc:1905] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:53.268237 8539 ts_tablet_manager.cc:1905] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:30:53.272493 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 8032
I20260926 21:30:53.274823 8541 ts_tablet_manager.cc:1918] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:53.274951 8541 log.cc:1196] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/c67b84efbf0c4767ba539324e57a64cf
I20260926 21:30:53.275393 8541 ts_tablet_manager.cc:1939] T c67b84efbf0c4767ba539324e57a64cf P 5099d51479c64cc9a3575f1ef3103e80: Deleting consensus metadata
I20260926 21:30:53.275523 8540 ts_tablet_manager.cc:1918] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:53.275657 8540 log.cc:1196] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/c67b84efbf0c4767ba539324e57a64cf
I20260926 21:30:53.276037 8540 ts_tablet_manager.cc:1939] T c67b84efbf0c4767ba539324e57a64cf P 53621701c7d544b18cc974524b4362d7: Deleting consensus metadata
I20260926 21:30:53.276599 7946 catalog_manager.cc:4841] TS 5099d51479c64cc9a3575f1ef3103e80 (127.0.97.2:43587): tablet c67b84efbf0c4767ba539324e57a64cf (table test-table [id=7e4363baac514e94bb7b0252db0887e8]) successfully deleted
I20260926 21:30:53.277686 7948 catalog_manager.cc:4841] TS 53621701c7d544b18cc974524b4362d7 (127.0.97.3:46561): tablet c67b84efbf0c4767ba539324e57a64cf (table test-table [id=7e4363baac514e94bb7b0252db0887e8]) successfully deleted
I20260926 21:30:53.282496 8539 ts_tablet_manager.cc:1918] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:30:53.282709 8539 log.cc:1196] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/c67b84efbf0c4767ba539324e57a64cf
I20260926 21:30:53.283183 8539 ts_tablet_manager.cc:1939] T c67b84efbf0c4767ba539324e57a64cf P 49574611798d4fc2a9b902ea838e9ce7: Deleting consensus metadata
I20260926 21:30:53.284258 7945 catalog_manager.cc:4841] TS 49574611798d4fc2a9b902ea838e9ce7 (127.0.97.1:44861): tablet c67b84efbf0c4767ba539324e57a64cf (table test-table [id=7e4363baac514e94bb7b0252db0887e8]) successfully deleted
Sep 26 21:30:53 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 21:30:53.291747 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:53.273765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.0.97.1:44861)
0926 21:30:53.276496 (+ 2731us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:53.276504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:53.276810 (+ 306us) server_negotiation.cc:407] Connection header received
0926 21:30:53.277110 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:53.277115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:53.277206 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:53.277292 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:53.278907 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.279774 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.280982 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.281432 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.281571 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:53.285720 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:53.285733 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:53.285749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:53.285779 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:53.288942 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:53.289595 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:53.289602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:53.289605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:53.289738 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:53.290134 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:53.290137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:53.290139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:53.290558 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:53.290887 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:53.291152 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 21:30:53.291369 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2599}
I20260926 21:30:53.292433 8163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:53.423470 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 8032
I20260926 21:30:53.450676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 8172
Sep 26 21:30:53 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 21:30:53.465142 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:53.451516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.0.97.2:43587)
0926 21:30:53.451725 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:53.451731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:53.451749 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:30:53.451876 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:53.451886 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:53.451943 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:53.452065 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:53.453209 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.453882 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.455204 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.455593 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.455721 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:53.459741 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:53.459756 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:53.459772 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:53.459806 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:53.462513 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:53.463153 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:53.463163 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:53.463167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:53.463289 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:53.463665 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:53.463670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:53.463674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:53.464015 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:53.464317 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:53.464593 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 21:30:53.464811 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:30:53.465750 8302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:53.606186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 8172
I20260926 21:30:53.632720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 8311
Sep 26 21:30:53 dist-test-slave-87bt krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 21:30:53.647046 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:53.633427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.0.97.3:46561)
0926 21:30:53.633897 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:53.633901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:53.633915 (+ 14us) server_negotiation.cc:407] Connection header received
0926 21:30:53.633965 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:53.633969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:53.634036 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:53.634150 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:53.635440 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.636127 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.637300 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:53.637700 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:53.637829 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:53.642006 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:53.642027 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:53.642043 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:53.642073 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:53.644637 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:53.645301 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:53.645307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:53.645309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:53.645453 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:53.645752 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:53.645755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:53.645757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:53.646055 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:53.646274 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:53.646591 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 21:30:53.646764 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":240,"threads_started":1}
I20260926 21:30:53.647722 8441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:53.768275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 8311
I20260926 21:30:53.791189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 7929
I20260926 21:30:53.792333 8001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:53.905616 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 7929
2026-09-26T21:30:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5869 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:54 dist-test-slave-87bt krb5kdc[8589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:54 dist-test-slave-87bt krb5kdc[8589](info): set up 2 sockets
Sep 26 21:30:54 dist-test-slave-87bt krb5kdc[8589](info): commencing operation
krb5kdc: starting...
W20260926 21:30:55.011529 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:55 dist-test-slave-87bt krb5kdc[8589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:55.237979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45253
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:55.571528 8602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:55.571893 8602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:55.572065 8602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:55.579326 8602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:55.579486 8602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:55.579581 8602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:55.579665 8602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:55.579758 8602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:55.589774 8602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:55.591176 8602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:55.592545 8602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:55.604509 8607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:55.604442 8608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:55.604465 8610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:55.605299 8602 server_base.cc:1034] running on GCE node
I20260926 21:30:55.606132 8602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:55.607708 8602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:55.608937 8602 hybrid_clock.cc:648] HybridClock initialized: now 1790458255608905 us; error 54 us; skew 500 ppm
I20260926 21:30:55.609280 8602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T21:30:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1769 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:30:55 dist-test-slave-87bt krb5kdc[8616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:30:55 dist-test-slave-87bt krb5kdc[8616](info): set up 2 sockets
Sep 26 21:30:55 dist-test-slave-87bt krb5kdc[8616](info): commencing operation
krb5kdc: starting...
W20260926 21:30:56.775965 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:30:56 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:30:56.993541 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44633
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:57.314728 8629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:57.315100 8629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:57.315277 8629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:57.322469 8629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:30:57.322643 8629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:57.322732 8629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:57.322844 8629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:30:57.322932 8629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:30:57.333076 8629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:57.334534 8629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:57.335902 8629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:57.348064 8635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:57.347997 8637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:57.348021 8634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:57.349190 8629 server_base.cc:1034] running on GCE node
I20260926 21:30:57.350157 8629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:57.351804 8629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:57.353060 8629 hybrid_clock.cc:648] HybridClock initialized: now 1790458257353008 us; error 48 us; skew 500 ppm
I20260926 21:30:57.353463 8629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:57 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:57.356935 8629 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:30:57.358546 8629 webserver.cc:460] Webserver started at http://127.0.97.62:44959/ using document root <none> and password file <none>
I20260926 21:30:57.359274 8629 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:57.359423 8629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:57.359743 8629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:57.362864 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "7df3d6a45121419b93ca945bce928872"
format_stamp: "Formatted at 2026-09-26 21:30:57 on dist-test-slave-87bt"
server_key: "47660b1d3c79274e006c3274a77d1096"
server_key_iv: "d1d8884ebf9ec1357a7f1bd281ea3efa"
server_key_version: "encryptionkey@0"
I20260926 21:30:57.363669 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "7df3d6a45121419b93ca945bce928872"
format_stamp: "Formatted at 2026-09-26 21:30:57 on dist-test-slave-87bt"
server_key: "47660b1d3c79274e006c3274a77d1096"
server_key_iv: "d1d8884ebf9ec1357a7f1bd281ea3efa"
server_key_version: "encryptionkey@0"
I20260926 21:30:57.368839 8629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:30:57.372282 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:57.373734 8629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:57.373929 8629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "7df3d6a45121419b93ca945bce928872"
format_stamp: "Formatted at 2026-09-26 21:30:57 on dist-test-slave-87bt"
server_key: "47660b1d3c79274e006c3274a77d1096"
server_key_iv: "d1d8884ebf9ec1357a7f1bd281ea3efa"
server_key_version: "encryptionkey@0"
I20260926 21:30:57.374135 8629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:57.396263 8629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:57.400547 8629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:57.400846 8629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:57.419605 8629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44633
I20260926 21:30:57.419634 8695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44633 every 8 connection(s)
I20260926 21:30:57.421006 8629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:30:57.423983 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 8629
I20260926 21:30:57.424243 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:30:57.424817 8696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:30:57.424880 388 external_mini_cluster.cc:1428] Setting key 6d4c213716530d642a46185e8d573abc
I20260926 21:30:57.439079 8696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Bootstrap starting.
Sep 26 21:30:57 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:57.442978 8696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Neither blocks nor log segments found. Creating new log.
I20260926 21:30:57.444289 8696 log.cc:824] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Log is configured to *not* fsync() on all Append() calls
I20260926 21:30:57.447597 8696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: No bootstrap required, opened a new log
I20260926 21:30:57.448848 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:57.426852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.0.97.62:44633)
0926 21:30:57.427653 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:57.427673 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:57.427722 (+ 49us) server_negotiation.cc:407] Connection header received
0926 21:30:57.429838 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:57.429868 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:57.430395 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:57.430875 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:57.432146 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:57.434057 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:57.435315 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:57.436408 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:57.436599 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:57.440377 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:57.440393 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:57.440413 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:57.440468 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:57.444343 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:57.445182 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:57.445189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:57.445195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:57.445353 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:57.445759 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:57.445763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:57.445766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:57.446406 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:57.446700 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:57.447107 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 21:30:57.447527 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":318,"threads_started":1}
I20260926 21:30:57.456127 8696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } }
I20260926 21:30:57.456461 8696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:30:57.456555 8696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df3d6a45121419b93ca945bce928872, State: Initialized, Role: FOLLOWER
I20260926 21:30:57.457088 8696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } }
I20260926 21:30:57.457415 8696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:30:57.457576 8696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:30:57.457777 8696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:30:57.460376 8696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } }
I20260926 21:30:57.460856 8696 leader_election.cc:302] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df3d6a45121419b93ca945bce928872; no voters:
I20260926 21:30:57.461444 8696 leader_election.cc:288] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:30:57.461623 8701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:30:57.462406 8701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 1 LEADER]: Becoming Leader. State: Replica: 7df3d6a45121419b93ca945bce928872, State: Running, Role: LEADER
I20260926 21:30:57.462957 8701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } }
I20260926 21:30:57.463523 8696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:30:57.465960 8702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7df3d6a45121419b93ca945bce928872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } } }
I20260926 21:30:57.466265 8702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:57.470670 8703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df3d6a45121419b93ca945bce928872. Latest consensus state: current_term: 1 leader_uuid: "7df3d6a45121419b93ca945bce928872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } } }
I20260926 21:30:57.471753 8703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: This master's current role is: LEADER
I20260926 21:30:57.472638 8709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:30:57.478304 8709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:30:57.488533 8709 catalog_manager.cc:1347] Generated new cluster ID: f785e771d78c4d6096a65d42fc387476
I20260926 21:30:57.488663 8709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:30:57.497298 8709 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:30:57.499203 8709 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:30:57.517872 8709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Generated new TSK 0
I20260926 21:30:57.519045 8709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:30:57.573508 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44633
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:57.900506 8722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:57.900790 8722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:57.900888 8722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:57.909267 8722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:57.909379 8722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:57.909490 8722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:30:57.919324 8722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:57.920521 8722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:57.921748 8722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:57.934696 8728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:57.934739 8730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:57.934774 8727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:57.935365 8722 server_base.cc:1034] running on GCE node
I20260926 21:30:57.935973 8722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:57.936879 8722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:57.938105 8722 hybrid_clock.cc:648] HybridClock initialized: now 1790458257938059 us; error 38 us; skew 500 ppm
I20260926 21:30:57.938534 8722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:57 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:57.946013 8722 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:30:57.947548 8722 webserver.cc:460] Webserver started at http://127.0.97.1:44977/ using document root <none> and password file <none>
I20260926 21:30:57.948271 8722 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:57.948421 8722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:57.948747 8722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:57.951742 8722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "3828345ace18403db9386a875d6994f7"
format_stamp: "Formatted at 2026-09-26 21:30:57 on dist-test-slave-87bt"
server_key: "3cf48a0450ebdea55ed0af82b95fa699"
server_key_iv: "6a3e5b35f82cca87de46616f956057ec"
server_key_version: "encryptionkey@0"
I20260926 21:30:57.952556 8722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3828345ace18403db9386a875d6994f7"
format_stamp: "Formatted at 2026-09-26 21:30:57 on dist-test-slave-87bt"
server_key: "3cf48a0450ebdea55ed0af82b95fa699"
server_key_iv: "6a3e5b35f82cca87de46616f956057ec"
server_key_version: "encryptionkey@0"
I20260926 21:30:57.957908 8722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:30:57.961465 8737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:57.962878 8722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:30:57.963052 8722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "3828345ace18403db9386a875d6994f7"
format_stamp: "Formatted at 2026-09-26 21:30:57 on dist-test-slave-87bt"
server_key: "3cf48a0450ebdea55ed0af82b95fa699"
server_key_iv: "6a3e5b35f82cca87de46616f956057ec"
server_key_version: "encryptionkey@0"
I20260926 21:30:57.963264 8722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:57.993460 8722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:57.997733 8722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:57.998034 8722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:57.999065 8722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:58.000414 8722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:58.000519 8722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:58.000664 8722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:58.000742 8722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:58.033882 8722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45185
I20260926 21:30:58.033927 8849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45185 every 8 connection(s)
I20260926 21:30:58.035322 8722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:30:58.035702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 8722
I20260926 21:30:58.035925 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:30:58.036737 388 external_mini_cluster.cc:1428] Setting key 16dea02e7ac1f48f74fa85a893758cb3
Sep 26 21:30:58 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:58.059725 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:58.038227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59959 (local address 127.0.97.62:44633)
0926 21:30:58.038749 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:58.038754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:58.040041 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 21:30:58.042988 (+ 2947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:58.042993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:58.043062 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:58.043148 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:58.045452 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:58.046166 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:58.047890 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:58.048654 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:58.048819 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:58.052366 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:58.052375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:58.052383 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:58.052431 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:58.056136 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:58.057074 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:58.057084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:58.057091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:58.057244 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:58.057681 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:58.057697 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:58.057703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:58.058106 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:58.058280 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:58.059040 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 21:30:58.059287 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":278,"threads_started":1}
I20260926 21:30:58.060436 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:58.039002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:59959)
0926 21:30:58.039848 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:58.039911 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:58.042653 (+ 2742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:58.043450 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:58.043465 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:58.044311 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:58.045174 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:58.045203 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:58.046338 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:58.046346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:58.047731 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:58.047743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:58.048889 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:58.048894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:58.049094 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:58.049806 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:58.049829 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:58.052107 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:58.056318 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:58.056326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:58.056341 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:58.056918 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:58.057364 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:58.057368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:58.057370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:58.057552 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:58.058237 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:58.058243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:58.058641 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 21:30:58.058878 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 21:30:58.059282 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":316,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:30:58.062038 8850 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44633
I20260926 21:30:58.062310 8850 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:58.063048 8850 heartbeater.cc:511] Master 127.0.97.62:44633 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:30:58.065441 8661 ts_manager.cc:194] Registered new tserver with Master: 3828345ace18403db9386a875d6994f7 (127.0.97.1:45185)
I20260926 21:30:58.067088 8661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:30:58.089807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44633
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:58.412014 8856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:58.412333 8856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:58.412501 8856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:58.419675 8856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:58.419811 8856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:58.419989 8856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:30:58.429514 8856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:58.430799 8856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:58.432032 8856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:58.444916 8861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:58.444892 8864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:58.444830 8862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:58.445677 8856 server_base.cc:1034] running on GCE node
I20260926 21:30:58.446273 8856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:58.447220 8856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:58.448412 8856 hybrid_clock.cc:648] HybridClock initialized: now 1790458258448350 us; error 64 us; skew 500 ppm
I20260926 21:30:58.448728 8856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:58 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:58.451948 8856 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:30:58.453352 8856 webserver.cc:460] Webserver started at http://127.0.97.2:39875/ using document root <none> and password file <none>
I20260926 21:30:58.454044 8856 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:58.454221 8856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:58.454581 8856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:58.457381 8856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "03d42c283f8e4cd9a106336ecd3e989c"
format_stamp: "Formatted at 2026-09-26 21:30:58 on dist-test-slave-87bt"
server_key: "61a62f7439a50b7b0eb3fd7554ea9604"
server_key_iv: "18eb564d5ea31ec28d28172a5dabe0b9"
server_key_version: "encryptionkey@0"
I20260926 21:30:58.458149 8856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "03d42c283f8e4cd9a106336ecd3e989c"
format_stamp: "Formatted at 2026-09-26 21:30:58 on dist-test-slave-87bt"
server_key: "61a62f7439a50b7b0eb3fd7554ea9604"
server_key_iv: "18eb564d5ea31ec28d28172a5dabe0b9"
server_key_version: "encryptionkey@0"
I20260926 21:30:58.463281 8856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:30:58.466673 8871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:58.468020 8856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:30:58.468204 8856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "03d42c283f8e4cd9a106336ecd3e989c"
format_stamp: "Formatted at 2026-09-26 21:30:58 on dist-test-slave-87bt"
server_key: "61a62f7439a50b7b0eb3fd7554ea9604"
server_key_iv: "18eb564d5ea31ec28d28172a5dabe0b9"
server_key_version: "encryptionkey@0"
I20260926 21:30:58.468400 8856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:58.501204 8856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:58.505671 8856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:58.505985 8856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:58.507001 8856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:58.508303 8856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:58.508409 8856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:58.508565 8856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:58.508643 8856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:58.541998 8856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34995
I20260926 21:30:58.542043 8983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34995 every 8 connection(s)
I20260926 21:30:58.543337 8856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:30:58.550320 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 8856
I20260926 21:30:58.550511 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:30:58.550935 388 external_mini_cluster.cc:1428] Setting key 4b8c055e138f21512499d75f7ec0bc2e
Sep 26 21:30:58 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:58.563803 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:58.545830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51967 (local address 127.0.97.62:44633)
0926 21:30:58.545995 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:58.546000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:58.547351 (+ 1351us) server_negotiation.cc:407] Connection header received
0926 21:30:58.549473 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:58.549477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:58.549541 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:58.549628 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:58.551410 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:58.552251 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:58.553969 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:58.554797 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:58.554944 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:58.558223 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:58.558232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:58.558239 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:58.558284 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:58.560898 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:58.561593 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:58.561599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:58.561605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:58.561729 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:58.562151 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:58.562162 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:58.562165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:58.562401 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:58.562586 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:58.563252 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 21:30:58.563463 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:30:58.564945 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:58.546336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:51967)
0926 21:30:58.547173 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:58.547223 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:58.549211 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:58.549858 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:58.549866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:58.550492 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:58.551231 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:58.551251 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:58.552412 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:58.552418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:58.553813 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:58.553825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:58.555000 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:58.555005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:58.555206 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:58.555878 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:58.555904 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:58.557987 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:58.561050 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:58.561058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:58.561075 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:58.561473 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:58.561837 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:58.561841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:58.561845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:58.562027 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:58.562561 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:58.562570 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:58.562896 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 21:30:58.563160 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 21:30:58.563703 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":322,"threads_started":1}
I20260926 21:30:58.566385 8984 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44633
I20260926 21:30:58.566754 8984 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:58.567632 8984 heartbeater.cc:511] Master 127.0.97.62:44633 requested a full tablet report, sending...
I20260926 21:30:58.569284 8661 ts_manager.cc:194] Registered new tserver with Master: 03d42c283f8e4cd9a106336ecd3e989c (127.0.97.2:34995)
I20260926 21:30:58.570065 8661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51967
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:30:58.597231 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44633
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:30:58.924315 8989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:30:58.924579 8989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:30:58.924672 8989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:30:58.931716 8989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:30:58.931825 8989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:30:58.931934 8989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:30:58.941267 8989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:30:58.942467 8989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:30:58.943759 8989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:30:58.956460 8995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:58.956578 8994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:30:58.956601 8997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:30:58.957333 8989 server_base.cc:1034] running on GCE node
I20260926 21:30:58.957944 8989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:30:58.958997 8989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:30:58.960192 8989 hybrid_clock.cc:648] HybridClock initialized: now 1790458258960132 us; error 56 us; skew 500 ppm
I20260926 21:30:58.960508 8989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:30:58 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:30:58.963822 8989 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:30:58.965312 8989 webserver.cc:460] Webserver started at http://127.0.97.3:37161/ using document root <none> and password file <none>
I20260926 21:30:58.966176 8989 fs_manager.cc:359] Metadata directory not provided
I20260926 21:30:58.966312 8989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:30:58.966765 8989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:30:58.969787 8989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "7ea89d4d791642c9b9ccd8be794f0fc2"
format_stamp: "Formatted at 2026-09-26 21:30:58 on dist-test-slave-87bt"
server_key: "e9369eb5cce722b831056c68d169b6a4"
server_key_iv: "3f894fd4ecfa6c9d5284ad020145b397"
server_key_version: "encryptionkey@0"
I20260926 21:30:58.970664 8989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7ea89d4d791642c9b9ccd8be794f0fc2"
format_stamp: "Formatted at 2026-09-26 21:30:58 on dist-test-slave-87bt"
server_key: "e9369eb5cce722b831056c68d169b6a4"
server_key_iv: "3f894fd4ecfa6c9d5284ad020145b397"
server_key_version: "encryptionkey@0"
I20260926 21:30:58.976639 8989 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 21:30:58.980374 9004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:58.981802 8989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:30:58.981999 8989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "7ea89d4d791642c9b9ccd8be794f0fc2"
format_stamp: "Formatted at 2026-09-26 21:30:58 on dist-test-slave-87bt"
server_key: "e9369eb5cce722b831056c68d169b6a4"
server_key_iv: "3f894fd4ecfa6c9d5284ad020145b397"
server_key_version: "encryptionkey@0"
I20260926 21:30:58.982221 8989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:30:58.997001 8989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:30:59.000808 8989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:30:59.001106 8989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:30:59.002033 8989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:30:59.003314 8989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:30:59.003417 8989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:59.003571 8989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:30:59.003647 8989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:30:59.036005 8989 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33425
I20260926 21:30:59.036108 9116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33425 every 8 connection(s)
I20260926 21:30:59.037549 8989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:30:59.047732 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 8989
I20260926 21:30:59.047909 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:30:59.048413 388 external_mini_cluster.cc:1428] Setting key c31cb49fe6cd08921b2f4642fb439c8e
Sep 26 21:30:59 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:30:59.057525 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:59.040340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57811 (local address 127.0.97.62:44633)
0926 21:30:59.040510 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:59.040514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:59.041797 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 21:30:59.043550 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:59.043555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:59.043623 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:59.043712 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:59.045188 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.045910 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.047697 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.048226 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.048325 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:59.052502 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:59.052507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:59.052511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:59.052542 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:59.055034 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:59.055680 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:59.055684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:59.055686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:59.055791 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:59.056178 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:59.056181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:59.056183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:59.056415 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:59.056522 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:59.057144 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 21:30:59.057293 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:30:59.058573 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:59.040906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:57811)
0926 21:30:59.041612 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 21:30:59.041669 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 21:30:59.043296 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:30:59.043913 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:30:59.043921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:30:59.044429 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:30:59.045012 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:59.045027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.046036 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.046040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:59.047522 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:30:59.047540 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.048457 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.048466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:30:59.048799 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:59.049582 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:30:59.049604 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:30:59.052301 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:30:59.055163 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:59.055170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:59.055183 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:59.055574 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:59.055879 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:30:59.055882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:30:59.055884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:30:59.056083 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:30:59.056513 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:30:59.056520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:30:59.056831 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 21:30:59.057051 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 21:30:59.057444 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":278,"threads_started":1}
I20260926 21:30:59.059908 9117 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44633
I20260926 21:30:59.060202 9117 heartbeater.cc:464] Registering TS with master...
I20260926 21:30:59.060851 9117 heartbeater.cc:511] Master 127.0.97.62:44633 requested a full tablet report, sending...
I20260926 21:30:59.062723 8661 ts_manager.cc:194] Registered new tserver with Master: 7ea89d4d791642c9b9ccd8be794f0fc2 (127.0.97.3:33425)
I20260926 21:30:59.063805 8661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57811
I20260926 21:30:59.064340 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:30:59.070600 8850 heartbeater.cc:503] Master 127.0.97.62:44633 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 21:30:59.203459 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 8722
Sep 26 21:30:59 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:30:59.220167 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:59.204645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.0.97.1:45185)
0926 21:30:59.205291 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:59.205298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:59.205325 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:30:59.205388 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:59.205401 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:59.205627 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:59.205767 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:59.207203 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.208292 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.209527 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.210129 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.210241 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:59.214351 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:59.214363 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:59.214378 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:59.214477 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:59.217326 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:59.218036 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:59.218044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:59.218046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:59.218169 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:59.218615 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:59.218622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:59.218625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:59.218962 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:59.219265 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:59.219553 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 21:30:59.219823 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":348,"threads_started":1}
I20260926 21:30:59.221436 8845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:59.303542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 8722
I20260926 21:30:59.320828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 8856
Sep 26 21:30:59 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:30:59.337296 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:59.321740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.0.97.2:34995)
0926 21:30:59.322319 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:59.322326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:59.322352 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:30:59.322416 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:59.322471 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:59.322697 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:59.322834 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:59.324006 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.325082 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.326350 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.327076 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.327198 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:59.331322 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:59.331337 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:59.331353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:59.331422 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:59.334317 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:59.335165 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:59.335173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:59.335175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:59.335317 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:59.335743 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:59.335747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:59.335749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:59.336157 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:59.336375 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:59.336743 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 21:30:59.336982 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":309,"threads_started":1}
I20260926 21:30:59.338828 8979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:59.420539 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 8856
I20260926 21:30:59.437346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 8989
Sep 26 21:30:59 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:30:59.453794 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:30:59.438341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.0.97.3:33425)
0926 21:30:59.439045 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 21:30:59.439052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:30:59.439084 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:30:59.439140 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:30:59.439146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:30:59.439355 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:30:59.439539 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:30:59.440794 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.441828 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.443148 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:30:59.443720 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:30:59.443860 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:30:59.448144 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:30:59.448156 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:30:59.448171 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:30:59.448205 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:30:59.451066 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:59.451746 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:59.451754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:59.451756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:59.451879 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:30:59.452247 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:30:59.452254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:30:59.452257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:30:59.452610 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:30:59.452939 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 21:30:59.453181 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 21:30:59.453443 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":376,"threads_started":1}
I20260926 21:30:59.455307 9112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:59.538367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 8989
I20260926 21:30:59.554122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 8629
I20260926 21:30:59.555384 8691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:30:59.655191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 8629
I20260926 21:30:59.675477 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44633
--webserver_interface=127.0.97.62
--webserver_port=44959
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:31:00.001727 9134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:00.002103 9134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:00.002280 9134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:00.009610 9134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:31:00.009778 9134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:00.009894 9134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:00.010008 9134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:31:00.010102 9134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:31:00.020151 9134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:00.021587 9134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:00.022962 9134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:00.034638 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:00.034582 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:00.034556 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:00.035261 9134 server_base.cc:1034] running on GCE node
I20260926 21:31:00.036279 9134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:00.037720 9134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:00.038964 9134 hybrid_clock.cc:648] HybridClock initialized: now 1790458260038892 us; error 35 us; skew 500 ppm
I20260926 21:31:00.039341 9134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:31:00.042675 9134 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:31:00.044204 9134 webserver.cc:460] Webserver started at http://127.0.97.62:44959/ using document root <none> and password file <none>
I20260926 21:31:00.044898 9134 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:00.045049 9134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:00.050575 9134 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:31:00.053395 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:00.054800 9134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:31:00.055004 9134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "7df3d6a45121419b93ca945bce928872"
format_stamp: "Formatted at 2026-09-26 21:30:57 on dist-test-slave-87bt"
server_key: "47660b1d3c79274e006c3274a77d1096"
server_key_iv: "d1d8884ebf9ec1357a7f1bd281ea3efa"
server_key_version: "encryptionkey@0"
I20260926 21:31:00.055975 9134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:00.069128 9134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:00.073534 9134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:00.073814 9134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:00.092610 9134 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44633
I20260926 21:31:00.092677 9200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44633 every 8 connection(s)
I20260926 21:31:00.094141 9134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:31:00.095885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 9134
I20260926 21:31:00.096822 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45185
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44633
--webserver_port=44977
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:31:00.104053 9201 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:31:00.107488 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Bootstrap starting.
I20260926 21:31:00.126475 9201 log.cc:824] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Log is configured to *not* fsync() on all Append() calls
I20260926 21:31:00.135020 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:31:00.135568 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Bootstrap complete.
I20260926 21:31:00.144580 9201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } }
I20260926 21:31:00.145390 9201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df3d6a45121419b93ca945bce928872, State: Initialized, Role: FOLLOWER
I20260926 21:31:00.146107 9201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } }
I20260926 21:31:00.146318 9201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:31:00.146493 9201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:31:00.146715 9201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:31:00.150504 9201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } }
I20260926 21:31:00.151047 9201 leader_election.cc:302] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df3d6a45121419b93ca945bce928872; no voters:
I20260926 21:31:00.151678 9201 leader_election.cc:288] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 21:31:00.151818 9205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:31:00.152899 9205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [term 2 LEADER]: Becoming Leader. State: Replica: 7df3d6a45121419b93ca945bce928872, State: Running, Role: LEADER
I20260926 21:31:00.153447 9205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } }
I20260926 21:31:00.153874 9201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:31:00.156148 9207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df3d6a45121419b93ca945bce928872. Latest consensus state: current_term: 2 leader_uuid: "7df3d6a45121419b93ca945bce928872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } } }
I20260926 21:31:00.156527 9207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: This master's current role is: LEADER
I20260926 21:31:00.156194 9206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7df3d6a45121419b93ca945bce928872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3d6a45121419b93ca945bce928872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44633 } } }
I20260926 21:31:00.156641 9206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872 [sys.catalog]: This master's current role is: LEADER
I20260926 21:31:00.157588 9212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:31:00.167138 9212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:31:00.168941 9212 catalog_manager.cc:1259] Loaded cluster ID: f785e771d78c4d6096a65d42fc387476
I20260926 21:31:00.169087 9212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:31:00.178356 9212 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:31:00.180235 9212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7df3d6a45121419b93ca945bce928872: Loaded TSK: 0
I20260926 21:31:00.182042 9212 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 21:31:00.450774 9203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:00.451100 9203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:00.451267 9203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:00.459231 9203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:00.459398 9203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:00.459594 9203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:31:00.470042 9203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44977
--tserver_master_addrs=127.0.97.62:44633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:00.471673 9203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:00.473095 9203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:00.488433 9228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:00.488433 9227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:00.489096 9230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:00.489791 9203 server_base.cc:1034] running on GCE node
I20260926 21:31:00.490558 9203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:00.491590 9203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:00.492810 9203 hybrid_clock.cc:648] HybridClock initialized: now 1790458260492748 us; error 57 us; skew 500 ppm
I20260926 21:31:00.493178 9203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:31:00.496659 9203 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 21:31:00.498338 9203 webserver.cc:460] Webserver started at http://127.0.97.1:44977/ using document root <none> and password file <none>
I20260926 21:31:00.499207 9203 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:00.499354 9203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:00.506585 9203 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:31:00.509577 9237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:00.511080 9203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:31:00.511261 9203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "3828345ace18403db9386a875d6994f7"
format_stamp: "Formatted at 2026-09-26 21:30:57 on dist-test-slave-87bt"
server_key: "3cf48a0450ebdea55ed0af82b95fa699"
server_key_iv: "6a3e5b35f82cca87de46616f956057ec"
server_key_version: "encryptionkey@0"
I20260926 21:31:00.512248 9203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:00.531291 9203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:00.535418 9203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:00.535697 9203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:00.536710 9203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:00.538045 9203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:00.538182 9203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:00.538328 9203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:00.538442 9203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:00.570842 9203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45185
I20260926 21:31:00.570902 9349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45185 every 8 connection(s)
I20260926 21:31:00.572409 9203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:31:00.582281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 9203
I20260926 21:31:00.583055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34995
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44633
--webserver_port=39875
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:00.592595 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:00.575645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:56097)
0926 21:31:00.576517 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:00.576561 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:00.578295 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:00.579845 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:00.579857 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:00.580517 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:00.581126 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:00.581146 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.583694 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.583699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:00.585177 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:00.585196 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.586884 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.586890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:00.587129 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:00.587882 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:00.587904 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:00.591612 (+ 3708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":392,"threads_started":1}
W20260926 21:31:00.593256 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:00.575456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56097 (local address 127.0.97.62:44633)
0926 21:31:00.576249 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:00.576267 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:00.576706 (+ 439us) server_negotiation.cc:407] Connection header received
0926 21:31:00.578927 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:00.578940 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:00.579303 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:00.579689 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:00.581361 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.583419 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.585384 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.586614 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.586835 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:00.592108 (+ 5273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56097: BlockingRecv error: recv got EOF from 127.0.97.1:56097 (error 108)
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":262,"threads_started":1}
W20260926 21:31:00.593434 9350 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44633 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44633: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:00.603718 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:00.594223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:39323)
0926 21:31:00.594443 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:00.594464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:00.594576 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:00.594875 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:00.594882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:00.595347 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:00.595559 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:00.595574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.596583 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.596592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:00.598300 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:00.598320 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.599039 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.599048 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:00.599277 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:00.599929 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:00.599960 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:00.603411 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 21:31:00.603870 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:00.594403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39323 (local address 127.0.97.62:44633)
0926 21:31:00.594582 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:00.594585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:00.594603 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:31:00.594677 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:00.594680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:00.594736 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:00.594817 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:00.595703 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.596421 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.598494 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.598878 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.598968 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:00.603643 (+ 4675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39323: BlockingRecv error: recv got EOF from 127.0.97.1:39323 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:00.613444 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:00.604480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:55199)
0926 21:31:00.604743 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:00.604762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:00.604871 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:00.605335 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:00.605338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:00.605600 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:00.605742 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:00.605750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.607377 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.607380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:00.608362 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:00.608374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.609275 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.609279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:00.609393 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:00.610014 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:00.610031 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:00.613144 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153}
W20260926 21:31:00.613579 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:00.604771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55199 (local address 127.0.97.62:44633)
0926 21:31:00.604986 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:00.604994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:00.605016 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:31:00.605080 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:00.605088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:00.605190 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:00.605310 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:00.605889 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.607237 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.608495 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:00.609138 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:00.609263 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:00.613327 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55199: BlockingRecv error: recv got EOF from 127.0.97.1:55199 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 21:31:00.613876 9350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:31:00.925495 9354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:00.925817 9354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:00.925987 9354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:00.933239 9354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:00.933405 9354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:00.933589 9354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:31:00.943820 9354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39875
--tserver_master_addrs=127.0.97.62:44633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:00.945142 9354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:00.946466 9354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:00.960139 9359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:00.960177 9360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:00.960851 9362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:00.961299 9354 server_base.cc:1034] running on GCE node
I20260926 21:31:00.961980 9354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:00.963028 9354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:00.964241 9354 hybrid_clock.cc:648] HybridClock initialized: now 1790458260964195 us; error 41 us; skew 500 ppm
I20260926 21:31:00.964567 9354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:31:00 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:31:00.967773 9354 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 21:31:00.969237 9354 webserver.cc:460] Webserver started at http://127.0.97.2:39875/ using document root <none> and password file <none>
I20260926 21:31:00.969929 9354 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:00.970080 9354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:00.975481 9354 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:31:00.978137 9369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:00.979545 9354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:31:00.979728 9354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "03d42c283f8e4cd9a106336ecd3e989c"
format_stamp: "Formatted at 2026-09-26 21:30:58 on dist-test-slave-87bt"
server_key: "61a62f7439a50b7b0eb3fd7554ea9604"
server_key_iv: "18eb564d5ea31ec28d28172a5dabe0b9"
server_key_version: "encryptionkey@0"
I20260926 21:31:00.980683 9354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:00.995878 9354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:00.999704 9354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:00.999981 9354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:01.000921 9354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:01.002209 9354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:01.002293 9354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:01.002458 9354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:01.002555 9354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 21:31:01.035296 9354 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34995
I20260926 21:31:01.035393 9481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34995 every 8 connection(s)
I20260926 21:31:01.036795 9354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:31:01.037959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 9354
I20260926 21:31:01.038669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33425
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44633
--webserver_port=37161
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 21:31:01.053742 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:01.039668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43799 (local address 127.0.97.62:44633)
0926 21:31:01.039847 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:01.039852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:01.041357 (+ 1505us) server_negotiation.cc:407] Connection header received
0926 21:31:01.043291 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:01.043297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:01.043385 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:01.043507 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:01.045027 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.045832 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.047539 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.048286 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.048451 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.053424 (+ 4973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43799: BlockingRecv error: recv got EOF from 127.0.97.2:43799 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:31:01.054247 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:01.040367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:43799)
0926 21:31:01.041181 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:01.041225 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:01.043007 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:01.043751 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:01.043759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:01.044271 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:01.044839 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.044856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.046003 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.046007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.047380 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.047394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.048503 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.048508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.048702 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.049451 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:01.049472 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:01.053154 (+ 3682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":335,"threads_started":1}
W20260926 21:31:01.055158 9482 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44633 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44633: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:01.064206 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:01.055837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:40667)
0926 21:31:01.055993 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:01.056010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:01.056105 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:01.056463 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:01.056466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:01.056756 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:01.056919 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.056926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.058478 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.058483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.059383 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.059393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.060511 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.060515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.060716 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.061323 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:01.061340 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:01.064013 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 21:31:01.064882 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:01.055937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40667 (local address 127.0.97.62:44633)
0926 21:31:01.056133 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:01.056140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:01.056163 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:31:01.056225 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:01.056233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:01.056319 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:01.056443 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:01.057065 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.058306 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.059515 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.060362 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.060520 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.064251 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40667: BlockingRecv error: recv got EOF from 127.0.97.2:40667 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:01.073807 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:01.064917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:51911)
0926 21:31:01.065159 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:01.065179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:01.065299 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:01.065780 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:01.065785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:01.066152 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:01.066326 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.066336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.067296 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.067303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.068834 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.068848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.069480 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.069486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.069654 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.070232 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:01.070253 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:01.073518 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260926 21:31:01.073839 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:01.065001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51911 (local address 127.0.97.62:44633)
0926 21:31:01.065158 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:01.065164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:01.065456 (+ 292us) server_negotiation.cc:407] Connection header received
0926 21:31:01.065524 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:01.065532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:01.065623 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:01.065737 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:01.066515 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.067174 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.068973 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.069348 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.069440 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.073676 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51911: BlockingRecv error: recv got EOF from 127.0.97.2:51911 (error 108)
Metrics: {"server-negotiator.queue_time_us":44,"thread_start_us":987,"threads_started":1}
W20260926 21:31:01.074100 9482 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:31:01.366891 9485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:01.367282 9485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:01.367456 9485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:01.374943 9485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:01.375099 9485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:01.375317 9485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:31:01.385571 9485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37161
--tserver_master_addrs=127.0.97.62:44633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:01.387079 9485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:01.388441 9485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:01.401471 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:01.401450 9492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:01.401564 9494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:01.408062 9485 server_base.cc:1034] running on GCE node
I20260926 21:31:01.408735 9485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:01.409667 9485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:01.410910 9485 hybrid_clock.cc:648] HybridClock initialized: now 1790458261410868 us; error 57 us; skew 500 ppm
I20260926 21:31:01.411222 9485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:31:01.414536 9485 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 21:31:01.416016 9485 webserver.cc:460] Webserver started at http://127.0.97.3:37161/ using document root <none> and password file <none>
I20260926 21:31:01.416710 9485 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:01.416859 9485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:01.421991 9485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:31:01.424549 9501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:01.425762 9485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 21:31:01.425926 9485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "7ea89d4d791642c9b9ccd8be794f0fc2"
format_stamp: "Formatted at 2026-09-26 21:30:58 on dist-test-slave-87bt"
server_key: "e9369eb5cce722b831056c68d169b6a4"
server_key_iv: "3f894fd4ecfa6c9d5284ad020145b397"
server_key_version: "encryptionkey@0"
I20260926 21:31:01.426908 9485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:01.453562 9485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:01.457698 9485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:01.457989 9485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:01.459061 9485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:01.460502 9485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:01.460628 9485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:01.460773 9485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:01.460867 9485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:01.494122 9485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33425
I20260926 21:31:01.494212 9613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33425 every 8 connection(s)
I20260926 21:31:01.495711 9485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:31:01.502696 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 9485
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 21:31:01.513826 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:01.498189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58119 (local address 127.0.97.62:44633)
0926 21:31:01.498388 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:01.498394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:01.499693 (+ 1299us) server_negotiation.cc:407] Connection header received
0926 21:31:01.501504 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:01.501509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:01.501573 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:01.501661 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:01.503343 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.504379 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.505945 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.506387 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.506584 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.513612 (+ 7028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58119: BlockingRecv error: recv got EOF from 127.0.97.3:58119 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:31:01.515040 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:01.498778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:58119)
0926 21:31:01.499531 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:01.499570 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:01.501243 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:01.501877 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:01.501885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:01.502497 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:01.503151 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.503169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.504554 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.504558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.505801 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.505812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.506689 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.506697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.507010 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.508884 (+ 1874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:01.508914 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:01.513507 (+ 4593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":290,"threads_started":1}
W20260926 21:31:01.515772 9614 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44633 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44633: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260926 21:31:01.519413 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:01.503923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.0.97.62:44633)
0926 21:31:01.504137 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:01.504142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:01.504309 (+ 167us) server_negotiation.cc:407] Connection header received
0926 21:31:01.504524 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:01.504536 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:01.504593 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:01.504702 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:01.506265 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.507452 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.508710 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.509363 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.509490 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.512110 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:01.512123 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:01.512146 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:01.512199 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:01.515936 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:01.516756 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:01.516765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:01.516774 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:01.516970 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:01.517391 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:01.517394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:01.517396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:01.517837 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:01.518180 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:01.518657 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 21:31:01.519027 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:01.525223 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:01.516479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:56887)
0926 21:31:01.516630 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:01.516648 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:01.516760 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:01.517297 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:01.517301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:01.517636 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:01.517767 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.517775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.519073 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.519087 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.520215 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.520228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.521118 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.521125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.521305 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.521880 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:01.521902 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:01.525027 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:31:01.525453 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:01.516559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56887 (local address 127.0.97.62:44633)
0926 21:31:01.516734 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:01.516740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:01.516757 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:31:01.516894 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:01.516897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:01.516948 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:01.517085 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:01.517906 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.518945 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.520357 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.520950 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.521055 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.525209 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56887: BlockingRecv error: recv got EOF from 127.0.97.3:56887 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:01.533442 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:01.526067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:38437)
0926 21:31:01.526231 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:01.526251 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:01.526336 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:01.526756 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:01.526759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:01.527025 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:01.527164 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.527171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.528220 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.528225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.529154 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.529165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.529785 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.529789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.529986 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.530633 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:01.530649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:01.533222 (+ 2573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 21:31:01.533645 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:01.526272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38437 (local address 127.0.97.62:44633)
0926 21:31:01.526517 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:01.526520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:01.526535 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:31:01.526573 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:01.526576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:01.526640 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:01.526715 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:01.527276 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.528060 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.529284 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.529651 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.529739 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.533403 (+ 3664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38437: BlockingRecv error: recv got EOF from 127.0.97.3:38437 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260926 21:31:01.533806 9614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:01 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:01.623282 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:01.614517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:37329)
0926 21:31:01.615096 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:01.615113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:01.615276 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:01.615642 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:01.615647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:01.616187 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:01.616397 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.616406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.617382 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.617389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.618337 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:01.618349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.619183 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.619189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:01.619361 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.620077 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:01.620105 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:01.623070 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":292,"threads_started":1}
W20260926 21:31:01.623466 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:01.614679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37329 (local address 127.0.97.62:44633)
0926 21:31:01.614900 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:01.614906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:01.615204 (+ 298us) server_negotiation.cc:407] Connection header received
0926 21:31:01.615430 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:01.615436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:01.615509 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:01.615615 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:01.616542 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.617236 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.618537 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:01.618997 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:01.619126 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:01.623258 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37329: BlockingRecv error: recv got EOF from 127.0.97.1:37329 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 21:31:02 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:02 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:02.083235 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:02.074628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:56221)
0926 21:31:02.075185 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:02.075201 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:02.075356 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:02.075687 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:02.075691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:02.076153 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:02.076352 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:02.076361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.077317 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.077326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:02.078280 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:02.078289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.079072 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.079079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:02.079235 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:02.079924 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:02.079940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:02.083022 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":323,"threads_started":1}
W20260926 21:31:02.083441 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:02.074732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56221 (local address 127.0.97.62:44633)
0926 21:31:02.074922 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:02.074930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:02.075288 (+ 358us) server_negotiation.cc:407] Connection header received
0926 21:31:02.075492 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:02.075496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:02.075552 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:02.075657 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:02.076500 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.077182 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.078450 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.078881 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.079013 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:02.083206 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56221: BlockingRecv error: recv got EOF from 127.0.97.2:56221 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 21:31:02 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:02 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:02.549219 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:02.534411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:43319)
0926 21:31:02.534901 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:02.534916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:02.535070 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:02.535419 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:02.535424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:02.535900 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:02.536076 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:02.536084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.537118 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.537125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:02.538065 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:02.538075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.538845 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.538852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:02.539014 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:02.545750 (+ 6736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:02.545768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:02.549002 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":243,"threads_started":1}
W20260926 21:31:02.549408 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:02.534511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43319 (local address 127.0.97.62:44633)
0926 21:31:02.534709 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:02.534717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:02.535005 (+ 288us) server_negotiation.cc:407] Connection header received
0926 21:31:02.535209 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:02.535213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:02.535283 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:02.535391 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:02.536219 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.536974 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.538203 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.538657 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.538782 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:02.549178 (+ 10396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43319: BlockingRecv error: recv got EOF from 127.0.97.3:43319 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 21:31:02 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:02 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:02.635614 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:02.624256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:56547)
0926 21:31:02.625065 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:02.625090 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:02.625202 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:02.625551 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:02.625559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:02.626069 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:02.626224 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:02.626233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.627480 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.627483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:02.629487 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:02.629506 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.630344 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.630352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:02.630790 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:02.631570 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:02.631598 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:02.635293 (+ 3695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":396,"threads_started":1}
W20260926 21:31:02.635875 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:02.624297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56547 (local address 127.0.97.62:44633)
0926 21:31:02.624518 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:02.624525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:02.625174 (+ 649us) server_negotiation.cc:407] Connection header received
0926 21:31:02.625306 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:02.625310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:02.625400 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:02.625482 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:02.626610 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.627352 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.629630 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:02.630169 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:02.630300 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:02.635572 (+ 5272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56547: BlockingRecv error: recv got EOF from 127.0.97.1:56547 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 21:31:03 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:03 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:03.101138 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:03.084354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:33305)
0926 21:31:03.084917 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:03.084935 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:03.085107 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:03.085534 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:03.085538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:03.086009 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:03.086170 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:03.086179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.087370 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.087376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:03.088340 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:03.088353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.089145 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.089152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:03.089327 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:03.097371 (+ 8044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:03.097397 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:03.100793 (+ 3396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
W20260926 21:31:03.101261 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:03.084515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33305 (local address 127.0.97.62:44633)
0926 21:31:03.084753 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:03.084760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:03.085030 (+ 270us) server_negotiation.cc:407] Connection header received
0926 21:31:03.085251 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:03.085257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:03.085356 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:03.085506 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:03.086345 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.087196 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.088500 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.088962 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.089126 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:03.101012 (+ 11886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33305: BlockingRecv error: recv got EOF from 127.0.97.2:33305 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 21:31:03 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:03 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:03.559020 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:03.550290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:54897)
0926 21:31:03.551006 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:03.551023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:03.551199 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:03.551590 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:03.551595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:03.552054 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:03.552241 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:03.552252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.553293 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.553300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:03.554331 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:03.554343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.555139 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.555146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:03.555315 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:03.555891 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:03.555908 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:03.558779 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":492,"threads_started":1}
W20260926 21:31:03.559330 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:03.550614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54897 (local address 127.0.97.62:44633)
0926 21:31:03.550855 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:03.550873 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:03.551126 (+ 253us) server_negotiation.cc:407] Connection header received
0926 21:31:03.551356 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:03.551360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:03.551434 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:03.551522 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:03.552398 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.553143 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.554502 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.554982 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.555118 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:03.558992 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54897: BlockingRecv error: recv got EOF from 127.0.97.3:54897 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 21:31:03 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:03 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:03.645067 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:03.636763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:54165)
0926 21:31:03.637366 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:03.637384 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:03.637560 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:03.637954 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:03.637957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:03.638310 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:03.638607 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:03.638618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.639813 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.639817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:03.640701 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:03.640711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.641378 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.641382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:03.641499 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:03.642133 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:03.642150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:03.644835 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":289,"threads_started":1}
W20260926 21:31:03.645347 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:03.636940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54165 (local address 127.0.97.62:44633)
0926 21:31:03.637155 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:03.637160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:03.637485 (+ 325us) server_negotiation.cc:407] Connection header received
0926 21:31:03.637718 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:03.637724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:03.637824 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:03.637937 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:03.638771 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.639667 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.640840 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:03.641253 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:03.641355 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:03.645049 (+ 3694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54165: BlockingRecv error: recv got EOF from 127.0.97.1:54165 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 21:31:04 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:04 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:04.110401 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:04.102168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:53377)
0926 21:31:04.102674 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:04.102692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:04.102855 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:04.103151 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:04.103154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:04.103480 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:04.103633 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:04.103640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.104585 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.104588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:04.105550 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:04.105562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.106273 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.106279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:04.106464 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:04.107188 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:04.107214 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:04.110047 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":299,"threads_started":1}
W20260926 21:31:04.110512 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:04.102323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53377 (local address 127.0.97.62:44633)
0926 21:31:04.102524 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:04.102528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:04.102790 (+ 262us) server_negotiation.cc:407] Connection header received
0926 21:31:04.102969 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:04.102972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:04.103031 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:04.103115 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:04.103762 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.104481 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.105694 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.106113 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.106222 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:04.110254 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53377: BlockingRecv error: recv got EOF from 127.0.97.2:53377 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 21:31:04 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:04 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:04.568526 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:04.560215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:34653)
0926 21:31:04.560764 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:04.560784 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:04.560944 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:04.561245 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:04.561248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:04.561555 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:04.561748 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:04.561756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.562786 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.562791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:04.563912 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:04.563923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.564782 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.564788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:04.564960 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:04.565502 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:04.565519 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:04.568176 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":278,"threads_started":1}
W20260926 21:31:04.568686 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:04.560267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34653 (local address 127.0.97.62:44633)
0926 21:31:04.560491 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:04.560500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:04.560880 (+ 380us) server_negotiation.cc:407] Connection header received
0926 21:31:04.561039 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:04.561043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:04.561114 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:04.561203 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:04.561874 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.562671 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.564045 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.564639 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.564772 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:04.568416 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34653: BlockingRecv error: recv got EOF from 127.0.97.3:34653 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 21:31:04 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:04 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:04.654312 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:04.646027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:53319)
0926 21:31:04.646682 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:04.646699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:04.646814 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:04.647126 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:04.647130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:04.647460 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:04.647606 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:04.647615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.648754 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.648761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:04.649760 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:04.649773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.650565 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.650571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:04.650767 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:04.651307 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:04.651324 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:04.653967 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":415,"threads_started":1}
W20260926 21:31:04.654472 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:04.646185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53319 (local address 127.0.97.62:44633)
0926 21:31:04.646359 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:04.646364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:04.646778 (+ 414us) server_negotiation.cc:407] Connection header received
0926 21:31:04.646918 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:04.646922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:04.646983 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:04.647071 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:04.647759 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.648615 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.649903 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:04.650391 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:04.650562 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:04.654185 (+ 3623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53319: BlockingRecv error: recv got EOF from 127.0.97.1:53319 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 21:31:05 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:05 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:05.119323 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:05.111360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:46579)
0926 21:31:05.111854 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:05.111869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:05.112026 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:05.112356 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:05.112361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:05.112731 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:05.112868 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:05.112876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.113828 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.113835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:05.114822 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:05.114833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.115578 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.115584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:05.115779 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:05.116326 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:05.116342 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:05.118986 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":264,"threads_started":1}
W20260926 21:31:05.119392 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:05.111440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46579 (local address 127.0.97.62:44633)
0926 21:31:05.111662 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:05.111670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:05.111955 (+ 285us) server_negotiation.cc:407] Connection header received
0926 21:31:05.112158 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:05.112164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:05.112226 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:05.112326 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:05.113009 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.113700 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.114961 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.115436 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.115571 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:05.119178 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46579: BlockingRecv error: recv got EOF from 127.0.97.2:46579 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 21:31:05 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:05 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:05.578336 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:05.569547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:56315)
0926 21:31:05.570067 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:05.570084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:05.570220 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:05.570661 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:05.570848 (+ 187us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:05.571230 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:05.571378 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:05.571386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.572794 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.572797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:05.573733 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:05.573743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.574527 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.574531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:05.574658 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:05.575251 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:05.575268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:05.578012 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":283,"threads_started":1}
W20260926 21:31:05.578507 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:05.569624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56315 (local address 127.0.97.62:44633)
0926 21:31:05.569792 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:05.569797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:05.570173 (+ 376us) server_negotiation.cc:407] Connection header received
0926 21:31:05.570337 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:05.570341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:05.570480 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:05.570611 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:05.571518 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.572680 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.573912 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.574326 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.574502 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:05.578223 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56315: BlockingRecv error: recv got EOF from 127.0.97.3:56315 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 21:31:05 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:05 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:05.664191 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:05.655485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:38281)
0926 21:31:05.656008 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:05.656026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:05.656143 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:05.656429 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:05.656433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:05.656851 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:05.657038 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:05.657046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.658015 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.658020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:05.659233 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:05.659246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.659981 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.659986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:05.660124 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:05.660729 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:05.660746 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:05.663931 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":258,"threads_started":1}
W20260926 21:31:05.664336 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:05.655556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38281 (local address 127.0.97.62:44633)
0926 21:31:05.655783 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:05.655789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:05.656110 (+ 321us) server_negotiation.cc:407] Connection header received
0926 21:31:05.656234 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:05.656238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:05.656293 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:05.656372 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:05.657167 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.657869 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.659356 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:05.659783 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:05.659888 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:05.664148 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38281: BlockingRecv error: recv got EOF from 127.0.97.1:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 21:31:06 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:06 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:06.129052 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:06.120314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:37853)
0926 21:31:06.120859 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:06.120876 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:06.121036 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:06.121363 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:06.121368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:06.121792 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:06.121945 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:06.121954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.123075 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.123081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:06.124011 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:06.124022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.124763 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.124769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:06.124941 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:06.125639 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:06.125655 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:06.128818 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
W20260926 21:31:06.129213 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:06.120535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37853 (local address 127.0.97.62:44633)
0926 21:31:06.120759 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:06.120765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:06.120958 (+ 193us) server_negotiation.cc:407] Connection header received
0926 21:31:06.121162 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:06.121168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:06.121232 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:06.121336 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:06.122102 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.122933 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.124182 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.124613 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.124748 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:06.128985 (+ 4237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37853: BlockingRecv error: recv got EOF from 127.0.97.2:37853 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 21:31:06 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:06 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:06.587141 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:06.579276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:37583)
0926 21:31:06.579726 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:06.579742 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:06.579844 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:06.580135 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:06.580139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:06.580514 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:06.580673 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:06.580681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.581610 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.581613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:06.582520 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:06.582530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.583146 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.583149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:06.583259 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:06.583863 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:06.583879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:06.586802 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
W20260926 21:31:06.587265 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:06.579310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37583 (local address 127.0.97.62:44633)
0926 21:31:06.579547 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:06.579554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:06.579816 (+ 262us) server_negotiation.cc:407] Connection header received
0926 21:31:06.579933 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:06.579937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:06.580016 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:06.580095 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:06.580787 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.581494 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.582631 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.583029 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.583120 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:06.586999 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37583: BlockingRecv error: recv got EOF from 127.0.97.3:37583 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 21:31:06 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:06 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:06.673506 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:06.665237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:35625)
0926 21:31:06.665811 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:06.665853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:06.665972 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:06.666271 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:06.666274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:06.666698 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:06.666858 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:06.666866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.667854 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.667859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:06.668815 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:06.668827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.669482 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.669487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:06.669604 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:06.670173 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:06.670190 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:06.673294 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":353,"threads_started":1}
W20260926 21:31:06.673766 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:06.665379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35625 (local address 127.0.97.62:44633)
0926 21:31:06.665553 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:06.665558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:06.665928 (+ 370us) server_negotiation.cc:407] Connection header received
0926 21:31:06.666084 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:06.666088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:06.666146 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:06.666233 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:06.666985 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.667697 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.668937 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:06.669356 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:06.669441 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:06.673494 (+ 4053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35625: BlockingRecv error: recv got EOF from 127.0.97.1:35625 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 21:31:07 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:07 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:07.138139 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:07.130239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:34617)
0926 21:31:07.130811 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:07.130869 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:07.131023 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:07.131434 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:07.131440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:07.131968 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:07.132168 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:07.132177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.133135 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.133138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:07.134001 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:07.134010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.134619 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.134623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:07.134734 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:07.135274 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:07.135290 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:07.137930 (+ 2640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":325,"threads_started":1}
W20260926 21:31:07.138376 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:07.130308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34617 (local address 127.0.97.62:44633)
0926 21:31:07.130556 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:07.130565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:07.130959 (+ 394us) server_negotiation.cc:407] Connection header received
0926 21:31:07.131201 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:07.131213 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:07.131276 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:07.131355 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:07.132306 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.133027 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.134111 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.134494 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.134580 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:07.138138 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34617: BlockingRecv error: recv got EOF from 127.0.97.2:34617 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 21:31:07 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:07 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:07.596308 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:07.588228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:59153)
0926 21:31:07.588734 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:07.588750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:07.588865 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:07.589151 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:07.589154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:07.589512 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:07.589653 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:07.589661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.590740 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.590743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:07.591641 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:07.591651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.592450 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.592455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:07.592581 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:07.593153 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:07.593170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:07.596093 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":308,"threads_started":1}
W20260926 21:31:07.596560 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:07.588193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59153 (local address 127.0.97.62:44633)
0926 21:31:07.588375 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:07.588381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:07.588831 (+ 450us) server_negotiation.cc:407] Connection header received
0926 21:31:07.588957 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:07.588961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:07.589031 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:07.589110 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:07.589794 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.590625 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.591835 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.592281 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.592385 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:07.596324 (+ 3939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59153: BlockingRecv error: recv got EOF from 127.0.97.3:59153 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 21:31:07 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:07 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:07.682334 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:07.674464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:50741)
0926 21:31:07.675010 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:07.675027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:07.675139 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:07.675482 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:07.675485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:07.675860 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:07.676037 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:07.676045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.676956 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.676960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:07.677859 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:07.677869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.678573 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.678577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:07.678687 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:07.679303 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:07.679319 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:07.682102 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":226,"threads_started":1}
W20260926 21:31:07.682636 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:07.674522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50741 (local address 127.0.97.62:44633)
0926 21:31:07.674730 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:07.674740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:07.675114 (+ 374us) server_negotiation.cc:407] Connection header received
0926 21:31:07.675277 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:07.675280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:07.675359 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:07.675438 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:07.676152 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.676820 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.677972 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:07.678396 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:07.678556 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:07.682366 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50741: BlockingRecv error: recv got EOF from 127.0.97.1:50741 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 21:31:08 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:08 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:08.153119 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:08.139174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:37741)
0926 21:31:08.139992 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:08.140010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:08.140161 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:08.140572 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:08.140577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:08.141068 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:08.141233 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:08.141242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.142187 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.142190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:08.143112 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:08.143122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.143705 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.143709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:08.143824 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:08.149750 (+ 5926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:08.149768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:08.152875 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":422,"threads_started":1}
W20260926 21:31:08.153314 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:08.139398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37741 (local address 127.0.97.62:44633)
0926 21:31:08.139581 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:08.139587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:08.140097 (+ 510us) server_negotiation.cc:407] Connection header received
0926 21:31:08.140299 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:08.140303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:08.140405 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:08.140504 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:08.141354 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.142074 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.143222 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.143586 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.143672 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:08.153066 (+ 9394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37741: BlockingRecv error: recv got EOF from 127.0.97.2:37741 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 21:31:08 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:08 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:08.605366 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:08.597248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:36685)
0926 21:31:08.597843 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:08.597860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:08.597971 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:08.598262 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:08.598265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:08.598662 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:08.598838 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:08.598846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.599838 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.599841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:08.600726 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:08.600736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.601387 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.601395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:08.601544 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:08.602185 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:08.602207 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:08.605108 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":259,"threads_started":1}
W20260926 21:31:08.605476 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:08.597363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36685 (local address 127.0.97.62:44633)
0926 21:31:08.597521 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:08.597525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:08.597930 (+ 405us) server_negotiation.cc:407] Connection header received
0926 21:31:08.598063 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:08.598066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:08.598138 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:08.598217 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:08.598962 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.599714 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.600849 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.601253 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.601347 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:08.605287 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36685: BlockingRecv error: recv got EOF from 127.0.97.3:36685 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 21:31:08 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:08 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:08.691814 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:08.683406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:48585)
0926 21:31:08.683916 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:08.683931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:08.684044 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:08.684391 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:08.684394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:08.684778 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:08.684927 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:08.684934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.686219 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.686244 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:08.687313 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:08.687325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.688039 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.688043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:08.688156 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:08.688850 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:08.688867 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:08.691590 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":288,"threads_started":1}
W20260926 21:31:08.692082 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:08.683614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48585 (local address 127.0.97.62:44633)
0926 21:31:08.683777 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:08.683781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:08.684002 (+ 221us) server_negotiation.cc:407] Connection header received
0926 21:31:08.684142 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:08.684146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:08.684204 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:08.684297 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:08.685071 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.686066 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.687446 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:08.687887 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:08.687981 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:08.691808 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48585: BlockingRecv error: recv got EOF from 127.0.97.1:48585 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 21:31:09 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:09 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:09.162002 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:09.154039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:33769)
0926 21:31:09.154499 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:09.154514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:09.154660 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:09.154978 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:09.154981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:09.155350 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:09.155495 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:09.155502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.156563 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.156566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:09.157453 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:09.157463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.158106 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.158110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:09.158224 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:09.158800 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:09.158816 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:09.161670 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":256,"threads_started":1}
W20260926 21:31:09.162081 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:09.154124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33769 (local address 127.0.97.62:44633)
0926 21:31:09.154366 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:09.154372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:09.154593 (+ 221us) server_negotiation.cc:407] Connection header received
0926 21:31:09.154804 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:09.154807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:09.154865 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:09.154937 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:09.155607 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.156447 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.157565 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.157960 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.158046 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:09.161871 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33769: BlockingRecv error: recv got EOF from 127.0.97.2:33769 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 26 21:31:09 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:09 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:09.614867 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:09.606241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:51057)
0926 21:31:09.606760 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:09.606777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:09.606934 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:09.607285 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:09.607290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:09.607699 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:09.607839 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:09.607846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.608855 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.608863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:09.609781 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:09.609792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.610587 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.610594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:09.610771 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:09.611437 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:09.611482 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:09.614634 (+ 3152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":277,"threads_started":1}
W20260926 21:31:09.615100 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:09.606393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51057 (local address 127.0.97.62:44633)
0926 21:31:09.606627 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:09.606635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:09.606867 (+ 232us) server_negotiation.cc:407] Connection header received
0926 21:31:09.607057 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:09.607063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:09.607135 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:09.607241 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:09.607985 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.608712 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.609931 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.610367 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.610549 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:09.614822 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51057: BlockingRecv error: recv got EOF from 127.0.97.3:51057 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 21:31:09 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:09 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:09.701610 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:09.692977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:42317)
0926 21:31:09.693573 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:09.693589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:09.693713 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:09.694139 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:09.694142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:09.694545 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:09.694710 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:09.694718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.695742 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.695746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:09.696824 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:09.696836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.697587 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.697593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:09.697763 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:09.698394 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:09.698411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:09.701269 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":334,"threads_started":1}
W20260926 21:31:09.701721 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:09.693032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42317 (local address 127.0.97.62:44633)
0926 21:31:09.693260 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:09.693265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:09.693702 (+ 437us) server_negotiation.cc:407] Connection header received
0926 21:31:09.693909 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:09.693914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:09.693981 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:09.694124 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:09.694863 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.695620 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.696971 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:09.697424 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:09.697554 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:09.701477 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42317: BlockingRecv error: recv got EOF from 127.0.97.1:42317 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 21:31:10 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:10 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:10.171698 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:10.163013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:39997)
0926 21:31:10.163538 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:10.163553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:10.163683 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:10.163980 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:10.163983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:10.164367 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:10.164532 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:10.164539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.165505 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.165509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:10.166571 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:10.166584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.167306 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.167311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:10.167437 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:10.168099 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:10.168117 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:10.171308 (+ 3191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":285,"threads_started":1}
W20260926 21:31:10.171736 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:10.163060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39997 (local address 127.0.97.62:44633)
0926 21:31:10.163241 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:10.163245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:10.163634 (+ 389us) server_negotiation.cc:407] Connection header received
0926 21:31:10.163788 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:10.163791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:10.163850 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:10.163937 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:10.164656 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.165370 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.166721 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.167137 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.167254 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:10.171449 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39997: BlockingRecv error: recv got EOF from 127.0.97.2:39997 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 21:31:10 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:10 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:10.623853 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:10.615739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:40155)
0926 21:31:10.616301 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:10.616316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:10.616471 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:10.616800 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:10.616804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:10.617210 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:10.617359 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:10.617367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.618372 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.618379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:10.619367 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:10.619377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.620115 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.620121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:10.620295 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:10.620922 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:10.620939 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:10.623614 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":310,"threads_started":1}
W20260926 21:31:10.624066 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:10.615798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40155 (local address 127.0.97.62:44633)
0926 21:31:10.616034 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:10.616062 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:10.616407 (+ 345us) server_negotiation.cc:407] Connection header received
0926 21:31:10.616600 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:10.616605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:10.616674 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:10.616772 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:10.617500 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.618237 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.619509 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.619956 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.620100 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:10.623832 (+ 3732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40155: BlockingRecv error: recv got EOF from 127.0.97.3:40155 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 21:31:10 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:10 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:10.711688 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:10.702783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:36877)
0926 21:31:10.703320 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:10.703342 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:10.703481 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:10.703818 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:10.703823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:10.704301 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:10.704442 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:10.704450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.705531 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.705537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:10.707058 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:10.707073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.707924 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.707929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:10.708108 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:10.708672 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:10.708689 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:10.711476 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":302,"threads_started":1}
W20260926 21:31:10.711962 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:10.702893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36877 (local address 127.0.97.62:44633)
0926 21:31:10.703123 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:10.703130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:10.703445 (+ 315us) server_negotiation.cc:407] Connection header received
0926 21:31:10.703605 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:10.703609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:10.703669 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:10.703788 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:10.704593 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.705366 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.707245 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:10.707766 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:10.707895 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:10.711661 (+ 3766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36877: BlockingRecv error: recv got EOF from 127.0.97.1:36877 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 21:31:11 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:11 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:11.181205 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:11.172854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:46347)
0926 21:31:11.173424 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:11.173441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:11.173564 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:11.173912 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:11.173916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:11.174279 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:11.174461 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:11.174470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.175445 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.175449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:11.176373 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:11.176384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.177079 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.177083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:11.177205 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:11.177895 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:11.177913 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:11.180984 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":318,"threads_started":1}
W20260926 21:31:11.181509 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:11.172928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46347 (local address 127.0.97.62:44633)
0926 21:31:11.173140 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:11.173144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:11.173547 (+ 403us) server_negotiation.cc:407] Connection header received
0926 21:31:11.173668 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:11.173672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:11.173735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:11.173826 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:11.174609 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.175314 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.176520 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.176916 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.177010 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:11.181180 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46347: BlockingRecv error: recv got EOF from 127.0.97.2:46347 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 21:31:11 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:11 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:11.633366 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:11.625067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:58641)
0926 21:31:11.625712 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:11.625732 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:11.625832 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:11.626200 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:11.626205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:11.626681 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:11.626833 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:11.626842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.627918 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.627924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:11.628837 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:11.628848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.629561 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.629566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:11.629752 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:11.630323 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:11.630339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:11.633037 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":382,"threads_started":1}
W20260926 21:31:11.633463 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:11.625260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58641 (local address 127.0.97.62:44633)
0926 21:31:11.625505 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:11.625512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:11.625814 (+ 302us) server_negotiation.cc:407] Connection header received
0926 21:31:11.625939 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:11.625944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:11.626027 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:11.626157 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:11.626972 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.627777 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.628971 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.629411 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.629553 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:11.633237 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58641: BlockingRecv error: recv got EOF from 127.0.97.3:58641 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 21:31:11 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:11 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:11.721338 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:11.712755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:43329)
0926 21:31:11.713273 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:11.713294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:11.713425 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:11.713842 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:11.713846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:11.714402 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:11.714629 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:11.714638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.715687 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.715694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:11.716677 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:11.716688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.717533 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.717539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:11.717720 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:11.718253 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:11.718271 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:11.720977 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
W20260926 21:31:11.721414 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:11.712831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43329 (local address 127.0.97.62:44633)
0926 21:31:11.713082 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:11.713088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:11.713386 (+ 298us) server_negotiation.cc:407] Connection header received
0926 21:31:11.713600 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:11.713608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:11.713687 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:11.713810 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:11.714764 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.715562 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.716811 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:11.717381 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:11.717503 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:11.721190 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43329: BlockingRecv error: recv got EOF from 127.0.97.1:43329 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 21:31:12 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:12 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:12.191244 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:12.182285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:57653)
0926 21:31:12.182807 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:12.182823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:12.182985 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:12.183324 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:12.183328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:12.183775 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:12.183954 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:12.183962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.185048 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.185054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:12.186060 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:12.186072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.186847 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.186854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:12.187034 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:12.187683 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:12.187699 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:12.190837 (+ 3138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":296,"threads_started":1}
W20260926 21:31:12.191358 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:12.182415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57653 (local address 127.0.97.62:44633)
0926 21:31:12.182658 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:12.182664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:12.182909 (+ 245us) server_negotiation.cc:407] Connection header received
0926 21:31:12.183124 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:12.183133 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:12.183193 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:12.183296 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:12.184099 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.184897 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.186205 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.186689 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.186823 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:12.191143 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57653: BlockingRecv error: recv got EOF from 127.0.97.2:57653 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 21:31:12 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:12 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:12.641942 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:12.634265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:35429)
0926 21:31:12.634774 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:12.634789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:12.634896 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:12.635184 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:12.635187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:12.635581 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:12.635746 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:12.635753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.636679 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.636683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:12.637567 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:12.637577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.638208 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.638211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:12.638327 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:12.638998 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:12.639015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:12.641711 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":303,"threads_started":1}
W20260926 21:31:12.642122 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:12.634367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35429 (local address 127.0.97.62:44633)
0926 21:31:12.634553 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:12.634557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:12.634858 (+ 301us) server_negotiation.cc:407] Connection header received
0926 21:31:12.635004 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:12.635007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:12.635064 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:12.635142 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:12.635861 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.636566 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.637695 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.638084 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.638175 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:12.641877 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35429: BlockingRecv error: recv got EOF from 127.0.97.3:35429 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 21:31:12 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:12 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:12.730576 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:12.722325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:36283)
0926 21:31:12.722843 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:12.722858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:12.723026 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:12.723361 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:12.723366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:12.723767 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:12.723914 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:12.723922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.725156 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.725163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:12.726065 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:12.726076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.726904 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.726910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:12.727082 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:12.727626 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:12.727642 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:12.730294 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":316,"threads_started":1}
W20260926 21:31:12.730855 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:12.722366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36283 (local address 127.0.97.62:44633)
0926 21:31:12.722592 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:12.722599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:12.722948 (+ 349us) server_negotiation.cc:407] Connection header received
0926 21:31:12.723156 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:12.723162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:12.723231 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:12.723330 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:12.724055 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.725024 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.726200 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:12.726757 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:12.726880 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:12.730558 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36283: BlockingRecv error: recv got EOF from 127.0.97.1:36283 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 21:31:13 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:13 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:13.206184 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:13.192346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:42187)
0926 21:31:13.192903 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:13.192920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:13.193101 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:13.193464 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:13.193468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:13.193859 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:13.194014 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:13.194022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.195133 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.195139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:13.196065 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:13.196077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.196792 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.196798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:13.196976 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:13.203114 (+ 6138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:13.203136 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:13.205951 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":304,"threads_started":1}
W20260926 21:31:13.206372 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:13.192432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42187 (local address 127.0.97.62:44633)
0926 21:31:13.192720 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:13.192726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:13.193025 (+ 299us) server_negotiation.cc:407] Connection header received
0926 21:31:13.193243 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:13.193247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:13.193308 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:13.193414 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:13.194163 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.194979 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.196213 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.196638 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.196775 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:13.206177 (+ 9402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42187: BlockingRecv error: recv got EOF from 127.0.97.2:42187 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 26 21:31:13 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:13 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:13.651240 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:13.643065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:59595)
0926 21:31:13.643814 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:13.643828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:13.643990 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:13.644317 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:13.644321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:13.644700 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:13.644847 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:13.644854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.645858 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.645863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:13.646822 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:13.646833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.647536 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.647541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:13.647709 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:13.648265 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:13.648281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:13.650913 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":521,"threads_started":1}
W20260926 21:31:13.651350 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:13.643231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59595 (local address 127.0.97.62:44633)
0926 21:31:13.643449 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:13.643458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:13.643917 (+ 459us) server_negotiation.cc:407] Connection header received
0926 21:31:13.644118 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:13.644124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:13.644189 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:13.644288 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:13.644988 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.645718 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.646973 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.647387 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.647512 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:13.651117 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59595: BlockingRecv error: recv got EOF from 127.0.97.3:59595 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 21:31:13 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:13 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:13.739935 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:13.731581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:58835)
0926 21:31:13.732120 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:13.732135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:13.732308 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:13.732650 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:13.732655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:13.733086 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:13.733249 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:13.733257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.734279 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.734284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:13.735280 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:13.735292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.736049 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.736055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:13.736227 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:13.736808 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:13.736825 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:13.739602 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":261,"threads_started":1}
W20260926 21:31:13.740023 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:13.731739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58835 (local address 127.0.97.62:44633)
0926 21:31:13.731999 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:13.732005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:13.732221 (+ 216us) server_negotiation.cc:407] Connection header received
0926 21:31:13.732439 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:13.732445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:13.732513 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:13.732618 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:13.733394 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.734131 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.735427 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:13.735861 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:13.736005 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:13.739793 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58835: BlockingRecv error: recv got EOF from 127.0.97.1:58835 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 21:31:14 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:14 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:14.215968 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:14.207289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:58897)
0926 21:31:14.207872 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:14.207889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:14.208059 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:14.208506 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:14.208511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:14.209071 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:14.209289 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:14.209299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.210484 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.210490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:14.211411 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:14.211423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.212064 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.212069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:14.212178 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:14.212824 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:14.212842 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:14.215736 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":295,"threads_started":1}
W20260926 21:31:14.216132 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:14.207623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58897 (local address 127.0.97.62:44633)
0926 21:31:14.207797 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:14.207803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:14.208013 (+ 210us) server_negotiation.cc:407] Connection header received
0926 21:31:14.208204 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:14.208210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:14.208284 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:14.208417 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:14.209458 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.210302 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.211537 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.211917 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.212005 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:14.215921 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58897: BlockingRecv error: recv got EOF from 127.0.97.2:58897 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 21:31:14 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:14 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:14.661036 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:14.652175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:51435)
0926 21:31:14.652741 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:14.652757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:14.652924 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:14.653255 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:14.653259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:14.653663 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:14.653864 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:14.653874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.655007 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.655013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:14.655936 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:14.655947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.656679 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.656685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:14.656859 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:14.657523 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:14.657542 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:14.660805 (+ 3263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":288,"threads_started":1}
W20260926 21:31:14.661294 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:14.652334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51435 (local address 127.0.97.62:44633)
0926 21:31:14.652561 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:14.652568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:14.652849 (+ 281us) server_negotiation.cc:407] Connection header received
0926 21:31:14.653047 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:14.653052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:14.653123 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:14.653227 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:14.654023 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.654873 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.656081 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.656494 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.656632 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:14.661033 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51435: BlockingRecv error: recv got EOF from 127.0.97.3:51435 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 21:31:14 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:14 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:14.749716 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:14.740983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:34153)
0926 21:31:14.741462 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:14.741479 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:14.741592 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:14.741896 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:14.741900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:14.742236 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:14.742402 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:14.742410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.743702 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.743709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:14.744951 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:14.744965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.746023 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.746028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:14.746166 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:14.746756 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:14.746774 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:14.749475 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":260,"threads_started":1}
W20260926 21:31:14.749927 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:14.741059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34153 (local address 127.0.97.62:44633)
0926 21:31:14.741299 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:14.741303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:14.741554 (+ 251us) server_negotiation.cc:407] Connection header received
0926 21:31:14.741696 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:14.741700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:14.741759 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:14.741853 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:14.742607 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.743504 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.745080 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:14.745486 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:14.745576 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:14.749677 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34153: BlockingRecv error: recv got EOF from 127.0.97.1:34153 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 26 21:31:15 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:15 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:15.225205 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:15.216865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:35277)
0926 21:31:15.217448 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:15.217463 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:15.217574 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:15.217842 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:15.217845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:15.218250 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:15.218414 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:15.218480 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.219434 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.219438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:15.220344 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:15.220354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.221027 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.221032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:15.221146 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:15.221734 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:15.221751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:15.224861 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":395,"threads_started":1}
W20260926 21:31:15.225324 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:15.216951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35277 (local address 127.0.97.62:44633)
0926 21:31:15.217182 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:15.217191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:15.217547 (+ 356us) server_negotiation.cc:407] Connection header received
0926 21:31:15.217655 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:15.217658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:15.217717 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:15.217802 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:15.218594 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.219317 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.220463 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.220906 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.220994 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:15.225097 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35277: BlockingRecv error: recv got EOF from 127.0.97.2:35277 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 21:31:15 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:15 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:15.670028 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:15.661871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.3:44413)
0926 21:31:15.662290 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:15.662307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:15.662461 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:15.662748 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:15.662751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:15.663074 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:15.663261 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:15.663269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.664242 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.664246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:15.665135 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:15.665146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.665834 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.665838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:15.665959 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:15.666722 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:15.666744 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:15.669784 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":229,"threads_started":1}
W20260926 21:31:15.670254 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:15.661906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44413 (local address 127.0.97.62:44633)
0926 21:31:15.662148 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:15.662155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:15.662386 (+ 231us) server_negotiation.cc:407] Connection header received
0926 21:31:15.662559 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:15.662563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:15.662624 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:15.662706 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:15.663412 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.664107 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.665269 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.665707 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.665802 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:15.670026 (+ 4224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44413: BlockingRecv error: recv got EOF from 127.0.97.3:44413 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 21:31:15 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:15 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:15.759334 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:15.750877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.1:37841)
0926 21:31:15.751394 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:15.751411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:15.751551 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:15.751908 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:15.751913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:15.752318 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:15.752574 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:15.752585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.753809 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.753817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:15.754864 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:15.754876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.755631 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.755636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:15.755811 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:15.756344 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:15.756360 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:15.759036 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":290,"threads_started":1}
W20260926 21:31:15.759446 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:15.750957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37841 (local address 127.0.97.62:44633)
0926 21:31:15.751150 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:15.751154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:15.751483 (+ 329us) server_negotiation.cc:407] Connection header received
0926 21:31:15.751684 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:15.751688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:15.751752 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:15.751836 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:15.752734 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.753670 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.755018 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:15.755493 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:15.755618 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:15.759240 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37841: BlockingRecv error: recv got EOF from 127.0.97.1:37841 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 21:31:16 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:31:16 dist-test-slave-87bt krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:31:16.234476 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:16.226414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44633 (local address 127.0.97.2:46099)
0926 21:31:16.227114 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:16.227132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:16.227258 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:16.227585 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:16.227589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:16.227958 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:16.228141 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:16.228149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.229103 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.229106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:16.230101 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:16.230112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.230850 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.230854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:16.230996 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:16.231607 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:16.231624 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:16.234185 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44633: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":397,"threads_started":1}
W20260926 21:31:16.234637 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:16.226572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46099 (local address 127.0.97.62:44633)
0926 21:31:16.226798 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:16.226806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:16.227219 (+ 413us) server_negotiation.cc:407] Connection header received
0926 21:31:16.227354 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:16.227358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:16.227417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:16.227498 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:16.228278 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.228983 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.230220 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.230720 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.230821 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:16.234353 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46099: BlockingRecv error: recv got EOF from 127.0.97.2:46099 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:31:16.513870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9203
I20260926 21:31:16.526089 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:16.515104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.0.97.1:45185)
0926 21:31:16.515683 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:16.515692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:16.515718 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:31:16.515776 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:16.515782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:16.516008 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:16.516167 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:16.517683 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.518776 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.519972 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.520532 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.520768 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:16.523116 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:16.523125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:16.523140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:16.523172 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:16.525577 (+ 2405us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 21:31:16.525872 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58694: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":314,"threads_started":1}
W20260926 21:31:16.526211 9663 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58694: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 21:31:16.526306 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9203: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45185: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 21:31:16.526651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 9203
I20260926 21:31:16.543401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9354
I20260926 21:31:16.555619 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:16.544452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.0.97.2:34995)
0926 21:31:16.545051 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:16.545058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:16.545092 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:31:16.545152 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:16.545158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:16.545388 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:16.545545 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:16.546852 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.547922 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.549122 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.549688 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.549785 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:16.552419 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:16.552428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:16.552442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:16.552478 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:16.555102 (+ 2624us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 21:31:16.555363 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52962: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":307,"threads_started":1}
W20260926 21:31:16.555732 9664 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52962: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 21:31:16.555827 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9354: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34995: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 21:31:16.555974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 9354
I20260926 21:31:16.572566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9485
I20260926 21:31:16.585843 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:16.573618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.0.97.3:33425)
0926 21:31:16.574226 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:16.574233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:16.574260 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:31:16.574319 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:16.574325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:16.574608 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:16.574767 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:16.576042 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.577126 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.578315 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:16.579019 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:16.579133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:16.581829 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:16.581838 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:16.581858 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:16.581900 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:16.585129 (+ 3229us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 21:31:16.585474 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37700: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":334,"threads_started":1}
W20260926 21:31:16.585883 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9485: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:33425: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 21:31:16.585963 9665 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37700: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 21:31:16.586040 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 9485
I20260926 21:31:16.602097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9134
I20260926 21:31:16.603420 9196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:16.717952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 9134
2026-09-26T21:31:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21044 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T21:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:31:16Z Disabled control of system clock
I20260926 21:31:16.793516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35763
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 21:31:17.122915 9670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:17.123263 9670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:17.123399 9670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:17.130811 9670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:31:17.130932 9670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:17.131018 9670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:17.131124 9670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:31:17.131196 9670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:31:17.140481 9670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:17.141868 9670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:17.143713 9670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:17.155835 9676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:17.155869 9678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:17.155941 9675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:17.156246 9670 server_base.cc:1034] running on GCE node
I20260926 21:31:17.157254 9670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:17.158807 9670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:17.160009 9670 hybrid_clock.cc:648] HybridClock initialized: now 1790458277159958 us; error 39 us; skew 500 ppm
I20260926 21:31:17.160378 9670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:31:17.162760 9670 webserver.cc:460] Webserver started at http://127.0.97.62:43639/ using document root <none> and password file <none>
I20260926 21:31:17.163479 9670 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:17.163622 9670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:17.163916 9670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:17.166862 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "33ef8d63a52d4807af44114aaa181b16"
format_stamp: "Formatted at 2026-09-26 21:31:17 on dist-test-slave-87bt"
server_key: "1c500dcecb57094d31cc11e54d92d396"
server_key_iv: "45efe6ca45c53f58db3386b6169e4dc9"
server_key_version: "encryptionkey@0"
I20260926 21:31:17.167642 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "33ef8d63a52d4807af44114aaa181b16"
format_stamp: "Formatted at 2026-09-26 21:31:17 on dist-test-slave-87bt"
server_key: "1c500dcecb57094d31cc11e54d92d396"
server_key_iv: "45efe6ca45c53f58db3386b6169e4dc9"
server_key_version: "encryptionkey@0"
I20260926 21:31:17.173062 9670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:31:17.176925 9684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:17.178527 9670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:31:17.178740 9670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "33ef8d63a52d4807af44114aaa181b16"
format_stamp: "Formatted at 2026-09-26 21:31:17 on dist-test-slave-87bt"
server_key: "1c500dcecb57094d31cc11e54d92d396"
server_key_iv: "45efe6ca45c53f58db3386b6169e4dc9"
server_key_version: "encryptionkey@0"
I20260926 21:31:17.178952 9670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:17.204700 9670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:17.205595 9670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:17.205884 9670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:17.223583 9670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35763
I20260926 21:31:17.223615 9735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35763 every 8 connection(s)
I20260926 21:31:17.224963 9670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:31:17.228566 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:31:17.235155 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 9670
I20260926 21:31:17.235324 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:31:17.235924 388 external_mini_cluster.cc:1428] Setting key 367a27e4e17d23671be63bcf67b8f9bc
I20260926 21:31:17.238883 9736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16: Bootstrap starting.
I20260926 21:31:17.242158 9736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16: Neither blocks nor log segments found. Creating new log.
I20260926 21:31:17.243276 9736 log.cc:824] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16: Log is configured to *not* fsync() on all Append() calls
I20260926 21:31:17.246213 9736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16: No bootstrap required, opened a new log
I20260926 21:31:17.249715 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:17.237882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.0.97.62:35763)
0926 21:31:17.238927 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:17.238944 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:17.238983 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:31:17.240674 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:17.240694 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:17.241144 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:17.241450 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:17.242123 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:17.243493 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:17.245138 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:17.245774 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:17.245902 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:17.247171 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:17.247178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:17.247192 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:17.247773 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:17.247857 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:17.247877 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:17.247982 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:17.248276 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 21:31:17.248681 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":405,"threads_started":1}
I20260926 21:31:17.254099 9736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef8d63a52d4807af44114aaa181b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35763 } }
I20260926 21:31:17.254455 9736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:31:17.254550 9736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ef8d63a52d4807af44114aaa181b16, State: Initialized, Role: FOLLOWER
I20260926 21:31:17.255178 9736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef8d63a52d4807af44114aaa181b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35763 } }
I20260926 21:31:17.255400 9736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:31:17.255523 9736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:31:17.255704 9736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:31:17.257820 9736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef8d63a52d4807af44114aaa181b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35763 } }
I20260926 21:31:17.258383 9736 leader_election.cc:302] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33ef8d63a52d4807af44114aaa181b16; no voters:
I20260926 21:31:17.258988 9736 leader_election.cc:288] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:31:17.259147 9741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:31:17.259821 9741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [term 1 LEADER]: Becoming Leader. State: Replica: 33ef8d63a52d4807af44114aaa181b16, State: Running, Role: LEADER
I20260926 21:31:17.260324 9741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef8d63a52d4807af44114aaa181b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35763 } }
I20260926 21:31:17.261054 9736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:31:17.263085 9743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ef8d63a52d4807af44114aaa181b16. Latest consensus state: current_term: 1 leader_uuid: "33ef8d63a52d4807af44114aaa181b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef8d63a52d4807af44114aaa181b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35763 } } }
I20260926 21:31:17.263505 9743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [sys.catalog]: This master's current role is: LEADER
I20260926 21:31:17.264575 9750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:31:17.264465 9742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33ef8d63a52d4807af44114aaa181b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ef8d63a52d4807af44114aaa181b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35763 } } }
I20260926 21:31:17.264983 9742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16 [sys.catalog]: This master's current role is: LEADER
I20260926 21:31:17.270807 9750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:31:17.279318 9750 catalog_manager.cc:1347] Generated new cluster ID: 74e2d863f5fe428fba953b24a4bf9ce9
I20260926 21:31:17.279490 9750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:31:17.292290 9750 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:31:17.293562 9750 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:31:17.302608 9750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33ef8d63a52d4807af44114aaa181b16: Generated new TSK 0
I20260926 21:31:17.303390 9750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:31:17.313512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35763
--builtin_ntp_servers=127.0.97.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 21:31:17.646307 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:17.646616 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:17.646701 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:17.654042 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:17.654151 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:17.654320 9760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:31:17.663679 9760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:17.664992 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:17.667217 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:17.679885 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:17.679963 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:17.679932 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:17.680786 9760 server_base.cc:1034] running on GCE node
I20260926 21:31:17.681332 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:17.682199 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:17.683406 9760 hybrid_clock.cc:648] HybridClock initialized: now 1790458277683302 us; error 72 us; skew 500 ppm
I20260926 21:31:17.683775 9760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:31:17.686029 9760 webserver.cc:460] Webserver started at http://127.0.97.1:45647/ using document root <none> and password file <none>
I20260926 21:31:17.686803 9760 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:17.686913 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:17.687227 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:17.689945 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "b28b53ae6ae64c9f980d8f35a5080d71"
format_stamp: "Formatted at 2026-09-26 21:31:17 on dist-test-slave-87bt"
server_key: "3284079a6c4c5d8c0f29cfa9a642ffbb"
server_key_iv: "a374f5f1d696c0708ec32e9669682504"
server_key_version: "encryptionkey@0"
I20260926 21:31:17.690739 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b28b53ae6ae64c9f980d8f35a5080d71"
format_stamp: "Formatted at 2026-09-26 21:31:17 on dist-test-slave-87bt"
server_key: "3284079a6c4c5d8c0f29cfa9a642ffbb"
server_key_iv: "a374f5f1d696c0708ec32e9669682504"
server_key_version: "encryptionkey@0"
I20260926 21:31:17.695585 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 21:31:17.698746 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:17.700011 9760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:31:17.700176 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "b28b53ae6ae64c9f980d8f35a5080d71"
format_stamp: "Formatted at 2026-09-26 21:31:17 on dist-test-slave-87bt"
server_key: "3284079a6c4c5d8c0f29cfa9a642ffbb"
server_key_iv: "a374f5f1d696c0708ec32e9669682504"
server_key_version: "encryptionkey@0"
I20260926 21:31:17.700379 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:17.717278 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:17.718057 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:17.718320 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:17.719220 9760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:17.720616 9760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:17.720731 9760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:17.720858 9760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:17.720945 9760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:17.753027 9760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37009
I20260926 21:31:17.753077 9886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37009 every 8 connection(s)
I20260926 21:31:17.754354 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:31:17.754910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 9760
I20260926 21:31:17.755111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:31:17.755975 388 external_mini_cluster.cc:1428] Setting key 18ae2db0466677a62503e5838c68d591
I20260926 21:31:17.758761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35763
--builtin_ntp_servers=127.0.97.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 21:31:17.767794 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:17.757119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42409 (local address 127.0.97.62:35763)
0926 21:31:17.757546 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:17.757552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:17.759084 (+ 1532us) server_negotiation.cc:407] Connection header received
0926 21:31:17.760825 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:17.760833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:17.760918 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:17.761045 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:17.762237 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:17.763046 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:17.764618 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:17.765309 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:17.765490 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:17.766764 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:17.766773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:17.766780 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:17.766806 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:17.766858 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:17.766876 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:17.766968 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:17.767219 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 21:31:17.767325 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":204,"threads_started":1}
I20260926 21:31:17.768604 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:17.757810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35763 (local address 127.0.97.1:42409)
0926 21:31:17.758934 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:17.758991 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:17.760533 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:17.761272 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:17.761286 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:17.761369 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:17.762000 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:17.762025 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:17.763207 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:17.763210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:17.764470 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:17.764485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:17.765576 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:17.765582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:17.765781 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:17.766491 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:31:17.766493 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:17.766533 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:31:17.766604 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:17.767012 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:17.767019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:17.767025 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:31:17.767165 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 21:31:17.767474 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":359,"threads_started":1}
I20260926 21:31:17.770053 9887 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35763
I20260926 21:31:17.770326 9887 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:17.771102 9887 heartbeater.cc:511] Master 127.0.97.62:35763 requested a full tablet report, sending...
I20260926 21:31:17.773083 9701 ts_manager.cc:194] Registered new tserver with Master: b28b53ae6ae64c9f980d8f35a5080d71 (127.0.97.1:37009)
I20260926 21:31:17.774688 9701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:42409
W20260926 21:31:18.085839 9891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:18.086138 9891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:18.086269 9891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:18.093556 9891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:18.093662 9891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:18.093827 9891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:31:18.103031 9891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:18.104322 9891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:18.106153 9891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:18.118918 9896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:18.118929 9897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:18.119218 9899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:18.119817 9891 server_base.cc:1034] running on GCE node
I20260926 21:31:18.120471 9891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:18.121429 9891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:18.122645 9891 hybrid_clock.cc:648] HybridClock initialized: now 1790458278122594 us; error 50 us; skew 500 ppm
I20260926 21:31:18.122967 9891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:31:18.125204 9891 webserver.cc:460] Webserver started at http://127.0.97.2:43001/ using document root <none> and password file <none>
I20260926 21:31:18.125913 9891 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:18.126039 9891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:18.126359 9891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:18.129177 9891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "951ffb4e01a04c6b806afb1edc67cd09"
format_stamp: "Formatted at 2026-09-26 21:31:18 on dist-test-slave-87bt"
server_key: "268c76be6134bf785eafc45c12728e76"
server_key_iv: "f3fd59ddfa5f047ce7fa8c3327ec02ff"
server_key_version: "encryptionkey@0"
I20260926 21:31:18.129997 9891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "951ffb4e01a04c6b806afb1edc67cd09"
format_stamp: "Formatted at 2026-09-26 21:31:18 on dist-test-slave-87bt"
server_key: "268c76be6134bf785eafc45c12728e76"
server_key_iv: "f3fd59ddfa5f047ce7fa8c3327ec02ff"
server_key_version: "encryptionkey@0"
I20260926 21:31:18.135349 9891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:31:18.138929 9905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:18.140339 9891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:31:18.140563 9891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "951ffb4e01a04c6b806afb1edc67cd09"
format_stamp: "Formatted at 2026-09-26 21:31:18 on dist-test-slave-87bt"
server_key: "268c76be6134bf785eafc45c12728e76"
server_key_iv: "f3fd59ddfa5f047ce7fa8c3327ec02ff"
server_key_version: "encryptionkey@0"
I20260926 21:31:18.140786 9891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:18.164141 9891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:18.165020 9891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:18.165318 9891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:18.166247 9891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:18.167802 9891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:18.167917 9891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:18.168054 9891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:18.168140 9891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:18.201812 9891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34865
I20260926 21:31:18.201853 10017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34865 every 8 connection(s)
I20260926 21:31:18.203132 9891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:31:18.212255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 9891
I20260926 21:31:18.212428 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:31:18.212904 388 external_mini_cluster.cc:1428] Setting key 0ca65c944b1e95527485ee763858a45c
I20260926 21:31:18.215634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35763
--builtin_ntp_servers=127.0.97.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 21:31:18.221503 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:18.205842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44513 (local address 127.0.97.62:35763)
0926 21:31:18.206062 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:18.206069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:18.207487 (+ 1418us) server_negotiation.cc:407] Connection header received
0926 21:31:18.209167 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:18.209172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:18.209246 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:18.209335 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:18.210539 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.211263 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.212852 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.213481 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.213590 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:18.220649 (+ 7059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:18.220658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:18.220666 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:18.220693 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:18.220739 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:18.220759 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:18.220869 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:18.221147 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 21:31:18.221233 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:31:18.223181 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:18.206458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35763 (local address 127.0.97.2:44513)
0926 21:31:18.207335 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:18.207375 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:18.208923 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:18.209560 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:18.209591 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:18.209722 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:18.210339 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:18.210355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.211392 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.211395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:18.212722 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:18.212734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.213704 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.213717 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:18.213966 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:18.220262 (+ 6296us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:31:18.220265 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:18.220314 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:31:18.220411 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:18.220884 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:18.220899 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:18.220911 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 21:31:18.221118 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 21:31:18.221646 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
I20260926 21:31:18.224918 10018 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35763
I20260926 21:31:18.225235 10018 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:18.226148 10018 heartbeater.cc:511] Master 127.0.97.62:35763 requested a full tablet report, sending...
I20260926 21:31:18.227773 9701 ts_manager.cc:194] Registered new tserver with Master: 951ffb4e01a04c6b806afb1edc67cd09 (127.0.97.2:34865)
I20260926 21:31:18.228901 9701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:44513
W20260926 21:31:18.545203 10021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:18.545537 10021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:18.545687 10021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:18.553104 10021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:18.553236 10021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:18.553400 10021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:31:18.562903 10021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:18.564213 10021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:18.566051 10021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:18.579502 10029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:18.579576 10026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:18.579540 10027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:18.580279 10021 server_base.cc:1034] running on GCE node
I20260926 21:31:18.580945 10021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:18.581866 10021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:18.583112 10021 hybrid_clock.cc:648] HybridClock initialized: now 1790458278583065 us; error 42 us; skew 500 ppm
I20260926 21:31:18.583503 10021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:31:18.585767 10021 webserver.cc:460] Webserver started at http://127.0.97.3:40365/ using document root <none> and password file <none>
I20260926 21:31:18.586544 10021 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:18.586671 10021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:18.586992 10021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:18.589847 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "5c15b94463a84575ac38509f512575f6"
format_stamp: "Formatted at 2026-09-26 21:31:18 on dist-test-slave-87bt"
server_key: "a53fcb0304a9cfc25dd3956a200cb843"
server_key_iv: "383b3eabe49e1f5adefca6271d1fd4c3"
server_key_version: "encryptionkey@0"
I20260926 21:31:18.590734 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5c15b94463a84575ac38509f512575f6"
format_stamp: "Formatted at 2026-09-26 21:31:18 on dist-test-slave-87bt"
server_key: "a53fcb0304a9cfc25dd3956a200cb843"
server_key_iv: "383b3eabe49e1f5adefca6271d1fd4c3"
server_key_version: "encryptionkey@0"
I20260926 21:31:18.595861 10021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 21:31:18.599212 10035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:18.600472 10021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:31:18.600638 10021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "5c15b94463a84575ac38509f512575f6"
format_stamp: "Formatted at 2026-09-26 21:31:18 on dist-test-slave-87bt"
server_key: "a53fcb0304a9cfc25dd3956a200cb843"
server_key_iv: "383b3eabe49e1f5adefca6271d1fd4c3"
server_key_version: "encryptionkey@0"
I20260926 21:31:18.600844 10021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:18.615358 10021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:18.616184 10021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:18.616467 10021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:18.617395 10021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:18.618989 10021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:18.619107 10021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:18.619234 10021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:18.619318 10021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:18.652994 10021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42791
I20260926 21:31:18.653051 10147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42791 every 8 connection(s)
I20260926 21:31:18.654285 10021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:31:18.659310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 10021
I20260926 21:31:18.659494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:31:18.660014 388 external_mini_cluster.cc:1428] Setting key 8f15e1292e83e5e877f9bf400a269269
I20260926 21:31:18.667021 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:18.656800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52913 (local address 127.0.97.62:35763)
0926 21:31:18.656982 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:18.656987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:18.658385 (+ 1398us) server_negotiation.cc:407] Connection header received
0926 21:31:18.660352 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:18.660359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:18.660456 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:18.660573 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:18.661974 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.662964 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.664535 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.664917 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.665017 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:18.666318 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:18.666322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:18.666325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:18.666340 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:18.666364 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:18.666374 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:18.666532 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:18.666705 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 21:31:18.666772 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:31:18.668098 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:18.657352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35763 (local address 127.0.97.3:52913)
0926 21:31:18.658250 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:18.658285 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:18.659983 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:18.660902 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:18.660912 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:18.661002 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:18.661786 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:18.661810 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.663122 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.663127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:18.664401 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:18.664412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.665133 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.665139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:18.665352 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:18.666060 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:31:18.666063 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:18.666101 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:31:18.666175 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:18.666530 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:18.666538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:18.666544 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:31:18.666686 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 21:31:18.666983 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
I20260926 21:31:18.669415 10148 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35763
I20260926 21:31:18.669691 10148 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:18.670394 10148 heartbeater.cc:511] Master 127.0.97.62:35763 requested a full tablet report, sending...
I20260926 21:31:18.671741 9701 ts_manager.cc:194] Registered new tserver with Master: 5c15b94463a84575ac38509f512575f6 (127.0.97.3:42791)
I20260926 21:31:18.672523 9701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52913
I20260926 21:31:18.675346 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 21:31:18.681434 9889 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:18.679835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.0.97.62:35763)
0926 21:31:18.680003 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:18.680007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:18.680509 (+ 502us) server_negotiation.cc:407] Connection header received
0926 21:31:18.680644 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:18.680648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:18.680715 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:18.680799 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:18.681217 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39318: BlockingRecv error: recv got EOF from 127.0.0.1:39318 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:31:18.682183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9760
I20260926 21:31:18.689550 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:18.683105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.0.97.1:37009)
0926 21:31:18.683582 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:18.683589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:18.683615 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:31:18.683666 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:18.683673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:18.683910 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:18.684047 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:18.684413 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.685410 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.686670 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.687246 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.687343 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:18.688246 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:18.688253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:18.688266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:18.688820 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:18.688860 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:18.688877 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:18.688971 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:18.689072 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 21:31:18.689180 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
I20260926 21:31:18.690754 9882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:18.771348 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 9760
I20260926 21:31:18.787284 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9891
I20260926 21:31:18.794644 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:18.788159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.0.97.2:34865)
0926 21:31:18.788676 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:18.788683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:18.788710 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:31:18.788765 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:18.788771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:18.788987 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:18.789110 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:18.789459 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.790500 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.791660 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.792190 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.792287 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:18.793204 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:18.793212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:18.793225 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:18.793821 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:18.793865 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:18.793883 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:18.793971 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:18.794109 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 21:31:18.794246 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
I20260926 21:31:18.795848 10013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:18.880069 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 9891
I20260926 21:31:18.895776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 10021
I20260926 21:31:18.903949 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:18.896747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.97.3:42791)
0926 21:31:18.897361 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:18.897368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:18.897399 (+ 31us) server_negotiation.cc:407] Connection header received
0926 21:31:18.897465 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:18.897472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:18.897720 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:18.897905 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:18.898370 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.899459 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.900685 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:18.901247 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:18.901379 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:18.902497 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:18.902514 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:18.902530 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:18.903115 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:18.903164 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:18.903183 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:18.903304 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:18.903455 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 21:31:18.903600 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":325,"threads_started":1}
I20260926 21:31:18.905272 10143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:18.989507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 10021
I20260926 21:31:19.005517 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 9670
I20260926 21:31:19.006825 9731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:19.106983 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 9670
2026-09-26T21:31:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2383 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T21:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:31:19Z Disabled control of system clock
I20260926 21:31:19.176965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35451
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 21:31:19.508174 10167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:19.508483 10167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:19.508659 10167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:19.516137 10167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:31:19.516289 10167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:19.516382 10167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:19.516489 10167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:31:19.516571 10167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:31:19.526480 10167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:19.527920 10167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:19.529685 10167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:19.541074 10172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:19.541177 10173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:19.541320 10175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:19.541978 10167 server_base.cc:1034] running on GCE node
I20260926 21:31:19.542860 10167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:19.544226 10167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:19.545444 10167 hybrid_clock.cc:648] HybridClock initialized: now 1790458279545408 us; error 35 us; skew 500 ppm
I20260926 21:31:19.545820 10167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:31:19.548238 10167 webserver.cc:460] Webserver started at http://127.0.97.62:35991/ using document root <none> and password file <none>
I20260926 21:31:19.548919 10167 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:19.549036 10167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:19.549371 10167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:19.552335 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "95e331b3dfce477e9bdb11164e1e3634"
format_stamp: "Formatted at 2026-09-26 21:31:19 on dist-test-slave-87bt"
server_key: "2ce12a515d274d1f0c5fce60a8ad2d2d"
server_key_iv: "67c6a1f89880c2b8ca110475bfd525de"
server_key_version: "encryptionkey@0"
I20260926 21:31:19.553110 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "95e331b3dfce477e9bdb11164e1e3634"
format_stamp: "Formatted at 2026-09-26 21:31:19 on dist-test-slave-87bt"
server_key: "2ce12a515d274d1f0c5fce60a8ad2d2d"
server_key_iv: "67c6a1f89880c2b8ca110475bfd525de"
server_key_version: "encryptionkey@0"
I20260926 21:31:19.558154 10167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:31:19.561611 10181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:19.563019 10167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:31:19.563200 10167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "95e331b3dfce477e9bdb11164e1e3634"
format_stamp: "Formatted at 2026-09-26 21:31:19 on dist-test-slave-87bt"
server_key: "2ce12a515d274d1f0c5fce60a8ad2d2d"
server_key_iv: "67c6a1f89880c2b8ca110475bfd525de"
server_key_version: "encryptionkey@0"
I20260926 21:31:19.563421 10167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:19.569367 10167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:19.570156 10167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:19.570521 10167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:19.588413 10167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35451
I20260926 21:31:19.588435 10232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35451 every 8 connection(s)
I20260926 21:31:19.589762 10167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:31:19.593477 10233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:31:19.598858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 10167
I20260926 21:31:19.599062 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:31:19.599697 388 external_mini_cluster.cc:1428] Setting key 06cb007b770d67352675e44a82870707
I20260926 21:31:19.604200 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634: Bootstrap starting.
I20260926 21:31:19.607880 10233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634: Neither blocks nor log segments found. Creating new log.
I20260926 21:31:19.608203 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:19.601763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.0.97.62:35451)
0926 21:31:19.602822 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:19.602835 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:19.602879 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:31:19.604535 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:19.604559 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:19.605005 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:19.605352 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:19.606563 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:19.606569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:19.606583 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:19.606597 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:19.606656 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:19.606672 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:19.606835 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:19.606956 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 21:31:19.607245 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":392,"threads_started":1}
I20260926 21:31:19.609229 10233 log.cc:824] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634: Log is configured to *not* fsync() on all Append() calls
I20260926 21:31:19.612291 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634: No bootstrap required, opened a new log
I20260926 21:31:19.619333 10233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e331b3dfce477e9bdb11164e1e3634" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } }
I20260926 21:31:19.619674 10233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:31:19.619793 10233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e331b3dfce477e9bdb11164e1e3634, State: Initialized, Role: FOLLOWER
I20260926 21:31:19.620354 10233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e331b3dfce477e9bdb11164e1e3634" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } }
I20260926 21:31:19.620566 10233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:31:19.620661 10233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:31:19.620783 10233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:31:19.623278 10233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e331b3dfce477e9bdb11164e1e3634" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } }
I20260926 21:31:19.623737 10233 leader_election.cc:302] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95e331b3dfce477e9bdb11164e1e3634; no voters:
I20260926 21:31:19.624293 10233 leader_election.cc:288] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:31:19.624470 10238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:31:19.625239 10238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [term 1 LEADER]: Becoming Leader. State: Replica: 95e331b3dfce477e9bdb11164e1e3634, State: Running, Role: LEADER
I20260926 21:31:19.625828 10238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e331b3dfce477e9bdb11164e1e3634" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } }
I20260926 21:31:19.626560 10233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:31:19.628844 10240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e331b3dfce477e9bdb11164e1e3634. Latest consensus state: current_term: 1 leader_uuid: "95e331b3dfce477e9bdb11164e1e3634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e331b3dfce477e9bdb11164e1e3634" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } } }
I20260926 21:31:19.629290 10240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [sys.catalog]: This master's current role is: LEADER
I20260926 21:31:19.630762 10239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95e331b3dfce477e9bdb11164e1e3634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e331b3dfce477e9bdb11164e1e3634" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } } }
I20260926 21:31:19.631242 10239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634 [sys.catalog]: This master's current role is: LEADER
I20260926 21:31:19.632324 10247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:31:19.637737 10247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:31:19.647321 10247 catalog_manager.cc:1347] Generated new cluster ID: 68564e7be3264a7ebf973d45fee1d15e
I20260926 21:31:19.647460 10247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:31:19.665823 10247 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:31:19.666244 10247 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:31:19.678814 10247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95e331b3dfce477e9bdb11164e1e3634: Generated new TSK 0
I20260926 21:31:19.679719 10247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:31:19.696290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35451
--builtin_ntp_servers=127.0.97.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 21:31:20.036365 10257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:20.036696 10257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:20.036854 10257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:20.044135 10257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:20.044273 10257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:20.044449 10257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:31:20.053714 10257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:20.055075 10257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:20.056908 10257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:20.069867 10262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:20.070019 10263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:20.069837 10265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:20.070964 10257 server_base.cc:1034] running on GCE node
I20260926 21:31:20.071537 10257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:20.072432 10257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:20.073763 10257 hybrid_clock.cc:648] HybridClock initialized: now 1790458280073715 us; error 50 us; skew 500 ppm
I20260926 21:31:20.074066 10257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:31:20.076193 10257 webserver.cc:460] Webserver started at http://127.0.97.1:39897/ using document root <none> and password file <none>
I20260926 21:31:20.076821 10257 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:20.076905 10257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:20.077152 10257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:20.081357 10257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "d08b7cb9ef7843758b2fbd8970a82534"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "06f98aad4f72ad0491697a64b7dfc1a0"
server_key_iv: "8537f9b2efa4e5fe4cbf27c05466df71"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.082067 10257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d08b7cb9ef7843758b2fbd8970a82534"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "06f98aad4f72ad0491697a64b7dfc1a0"
server_key_iv: "8537f9b2efa4e5fe4cbf27c05466df71"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.087041 10257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:31:20.090317 10271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:20.091617 10257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:31:20.091778 10257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "d08b7cb9ef7843758b2fbd8970a82534"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "06f98aad4f72ad0491697a64b7dfc1a0"
server_key_iv: "8537f9b2efa4e5fe4cbf27c05466df71"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.092001 10257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:20.097398 10257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:20.098168 10257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:20.098491 10257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:20.099375 10257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:20.100737 10257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:20.100855 10257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:20.100986 10257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:20.101068 10257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:20.132910 10257 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40813
I20260926 21:31:20.132977 10383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40813 every 8 connection(s)
I20260926 21:31:20.134207 10257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:31:20.141302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 10257
I20260926 21:31:20.141510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:31:20.142107 388 external_mini_cluster.cc:1428] Setting key 2cd3a0876558872ebb43504e9df5eb8a
I20260926 21:31:20.142554 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:20.136825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57937 (local address 127.0.97.62:35451)
0926 21:31:20.137224 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:20.137228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:20.138473 (+ 1245us) server_negotiation.cc:407] Connection header received
0926 21:31:20.140155 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:20.140161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:20.140241 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:20.140368 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:20.141758 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:20.141764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:20.141768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:20.141786 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:20.141831 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:20.141842 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:20.141960 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:20.142166 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 21:31:20.142237 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":214,"threads_started":1}
I20260926 21:31:20.143673 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:20.137425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35451 (local address 127.0.97.1:57937)
0926 21:31:20.138304 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:20.138339 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:20.139889 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:20.140610 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:20.140619 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:20.140717 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:20.141422 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:31:20.141426 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:20.141476 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:31:20.141585 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:20.141932 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:20.141943 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:20.141950 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 21:31:20.142249 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 21:31:20.142551 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":274,"threads_started":1}
I20260926 21:31:20.145040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35451
--builtin_ntp_servers=127.0.97.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 21:31:20.145289 10384 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35451
I20260926 21:31:20.145520 10384 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:20.146247 10384 heartbeater.cc:511] Master 127.0.97.62:35451 requested a full tablet report, sending...
I20260926 21:31:20.148437 10198 ts_manager.cc:194] Registered new tserver with Master: d08b7cb9ef7843758b2fbd8970a82534 (127.0.97.1:40813)
W20260926 21:31:20.476090 10388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:20.476382 10388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:20.476471 10388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:20.483903 10388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:20.484018 10388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:20.484138 10388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:31:20.493194 10388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:20.494407 10388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:20.496335 10388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:20.509656 10393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:20.509546 10394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:20.509537 10396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:20.510326 10388 server_base.cc:1034] running on GCE node
I20260926 21:31:20.511018 10388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:20.511988 10388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:20.513217 10388 hybrid_clock.cc:648] HybridClock initialized: now 1790458280513171 us; error 46 us; skew 500 ppm
I20260926 21:31:20.513557 10388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:31:20.515863 10388 webserver.cc:460] Webserver started at http://127.0.97.2:42065/ using document root <none> and password file <none>
I20260926 21:31:20.516553 10388 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:20.516680 10388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:20.516999 10388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:20.519934 10388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "0c9d4f01fcf74dacba41fd1fbbb7672f"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "23fda020eaa638dec13ce7e23a2d54c5"
server_key_iv: "3b1fca6825591006a0f73d1179048ba3"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.520715 10388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0c9d4f01fcf74dacba41fd1fbbb7672f"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "23fda020eaa638dec13ce7e23a2d54c5"
server_key_iv: "3b1fca6825591006a0f73d1179048ba3"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.525641 10388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:31:20.529404 10402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:20.530771 10388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:31:20.530930 10388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "0c9d4f01fcf74dacba41fd1fbbb7672f"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "23fda020eaa638dec13ce7e23a2d54c5"
server_key_iv: "3b1fca6825591006a0f73d1179048ba3"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.531157 10388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:20.537048 10388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:20.537896 10388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:20.538164 10388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:20.539172 10388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:20.540560 10388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:20.540681 10388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:20.540827 10388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:20.540911 10388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:20.573406 10388 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43135
I20260926 21:31:20.573449 10514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43135 every 8 connection(s)
I20260926 21:31:20.574782 10388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:31:20.578794 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 10388
I20260926 21:31:20.578992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:31:20.579576 388 external_mini_cluster.cc:1428] Setting key 09d78a0ac08c12f4eb16cdc810077eef
I20260926 21:31:20.582566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35451
--builtin_ntp_servers=127.0.97.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 21:31:20.583645 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:20.577263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35283 (local address 127.0.97.62:35451)
0926 21:31:20.577443 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:20.577447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:20.579081 (+ 1634us) server_negotiation.cc:407] Connection header received
0926 21:31:20.581150 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:20.581154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:20.581225 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:20.581325 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:20.582820 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:20.582824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:20.582828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:20.582843 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:20.582872 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:20.582883 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:20.582955 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:20.583228 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 21:31:20.583301 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:31:20.584465 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:20.577804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35451 (local address 127.0.97.2:35283)
0926 21:31:20.578878 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:20.578939 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:20.580872 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:20.581602 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:20.581621 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:20.581720 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:20.582475 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:31:20.582478 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:20.582525 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:31:20.582613 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:20.583025 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:20.583032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:20.583037 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:31:20.583253 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 21:31:20.583456 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":265,"threads_started":1}
I20260926 21:31:20.585912 10515 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35451
I20260926 21:31:20.586231 10515 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:20.586975 10515 heartbeater.cc:511] Master 127.0.97.62:35451 requested a full tablet report, sending...
I20260926 21:31:20.588685 10198 ts_manager.cc:194] Registered new tserver with Master: 0c9d4f01fcf74dacba41fd1fbbb7672f (127.0.97.2:43135)
W20260926 21:31:20.918653 10518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:20.918982 10518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:20.919140 10518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:20.926221 10518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:20.926364 10518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:20.926525 10518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:31:20.935786 10518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:20.937090 10518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:20.938736 10518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:20.951365 10524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:20.951421 10523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:20.951434 10526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:20.952353 10518 server_base.cc:1034] running on GCE node
I20260926 21:31:20.952912 10518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:20.953790 10518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:20.955000 10518 hybrid_clock.cc:648] HybridClock initialized: now 1790458280954953 us; error 58 us; skew 500 ppm
I20260926 21:31:20.955307 10518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:31:20.957366 10518 webserver.cc:460] Webserver started at http://127.0.97.3:42223/ using document root <none> and password file <none>
I20260926 21:31:20.958042 10518 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:20.958154 10518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:20.958485 10518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:20.962611 10518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "334dfed254894ce08d103f076176e470"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "ff79f351a3b3aff84b25d82a7cadfe63"
server_key_iv: "ccacfd06971cca9d6308f853a3691136"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.963661 10518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "334dfed254894ce08d103f076176e470"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "ff79f351a3b3aff84b25d82a7cadfe63"
server_key_iv: "ccacfd06971cca9d6308f853a3691136"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.970626 10518 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 21:31:20.975123 10532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:20.976702 10518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:31:20.976850 10518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "334dfed254894ce08d103f076176e470"
format_stamp: "Formatted at 2026-09-26 21:31:20 on dist-test-slave-87bt"
server_key: "ff79f351a3b3aff84b25d82a7cadfe63"
server_key_iv: "ccacfd06971cca9d6308f853a3691136"
server_key_version: "encryptionkey@0"
I20260926 21:31:20.977046 10518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:20.982650 10518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:20.983387 10518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:20.983609 10518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:20.984472 10518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:20.985803 10518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:20.985912 10518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:20.986034 10518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:20.986116 10518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:21.020355 10518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42177
I20260926 21:31:21.020407 10644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42177 every 8 connection(s)
I20260926 21:31:21.021651 10518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:31:21.029287 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:21.024127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45759 (local address 127.0.97.62:35451)
0926 21:31:21.024288 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:21.024293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:21.025595 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 21:31:21.027297 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:21.027326 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:21.027379 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:21.027452 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:21.028695 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:21.028699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:21.028702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:21.028717 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:21.028746 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:21.028763 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:21.028862 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:21.029042 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 21:31:21.029090 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:31:21.029588 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 10518
I20260926 21:31:21.029776 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:31:21.030304 388 external_mini_cluster.cc:1428] Setting key d553d97b899985d2610ff2005687d449
I20260926 21:31:21.030596 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:21.024572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35451 (local address 127.0.97.3:45759)
0926 21:31:21.025448 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:21.025482 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:21.027023 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:21.027721 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:21.027730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:21.027824 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:21.028427 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:31:21.028429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:21.028466 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:31:21.028546 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:21.028855 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:21.028867 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:21.028877 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:31:21.029095 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 21:31:21.029341 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":284,"threads_started":1}
I20260926 21:31:21.032020 10645 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35451
I20260926 21:31:21.032397 10645 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:21.033079 10645 heartbeater.cc:511] Master 127.0.97.62:35451 requested a full tablet report, sending...
I20260926 21:31:21.034466 10197 ts_manager.cc:194] Registered new tserver with Master: 334dfed254894ce08d103f076176e470 (127.0.97.3:42177)
I20260926 21:31:21.045528 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 21:31:21.051293 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:31:21.049799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.97.62:35451)
0926 21:31:21.050003 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:21.050011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:21.050491 (+ 480us) server_negotiation.cc:407] Connection header received
0926 21:31:21.050648 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:21.050651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:21.050701 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:21.050789 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:21.051058 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60240: BlockingRecv error: recv got EOF from 127.0.0.1:60240 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:31:21.051877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 10257
I20260926 21:31:21.055073 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:21.052771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.0.97.1:40813)
0926 21:31:21.053238 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:21.053245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:21.053270 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:31:21.053325 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:21.053332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:21.053543 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:21.053666 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:21.054521 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:21.054528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:21.054539 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:21.054552 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:21.054595 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:21.054613 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:21.054731 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:21.054786 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 21:31:21.054864 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260926 21:31:21.056224 10379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:21.142602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 10257
I20260926 21:31:21.168738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 10388
I20260926 21:31:21.172508 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:21.169768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.0.97.2:43135)
0926 21:31:21.170260 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:21.170267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:21.170293 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:31:21.170351 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:21.170358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:21.170637 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:21.170800 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:21.171812 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:21.171825 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:21.171838 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:21.171851 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:21.171900 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:21.171918 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:21.172052 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:21.172145 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 21:31:21.172264 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260926 21:31:21.173889 10510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:21.252768 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 10388
I20260926 21:31:21.276060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 10518
I20260926 21:31:21.280268 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:21.277108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.0.97.3:42177)
0926 21:31:21.277826 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:21.277833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:21.277863 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:31:21.277924 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:21.277930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:21.278138 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:21.278316 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:21.279545 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:21.279553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:21.279564 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:31:21.279583 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:21.279626 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:31:21.279641 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:21.279818 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:21.279889 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 21:31:21.279987 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":414,"threads_started":1}
I20260926 21:31:21.281642 10640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:21.361408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 10518
I20260926 21:31:21.377117 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 10167
I20260926 21:31:21.378374 10228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:21.477576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 10167
2026-09-26T21:31:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2370 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:31:21 dist-test-slave-87bt krb5kdc[10663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:31:21 dist-test-slave-87bt krb5kdc[10663](info): set up 2 sockets
Sep 26 21:31:21 dist-test-slave-87bt krb5kdc[10663](info): commencing operation
krb5kdc: starting...
W20260926 21:31:22.583324 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:31:22 dist-test-slave-87bt krb5kdc[10663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:31:22.822830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33067
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:31:23.146612 10676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:23.146921 10676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:23.147012 10676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:23.154197 10676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:31:23.154321 10676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:23.154367 10676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:23.154479 10676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:31:23.154538 10676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:31:23.164384 10676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:23.165763 10676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:23.167191 10676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:23.179286 10681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:23.179214 10684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:23.179200 10682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:23.180187 10676 server_base.cc:1034] running on GCE node
I20260926 21:31:23.181236 10676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:23.183084 10676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:23.184333 10676 hybrid_clock.cc:648] HybridClock initialized: now 1790458283184257 us; error 62 us; skew 500 ppm
I20260926 21:31:23.184734 10676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:31:23 dist-test-slave-87bt krb5kdc[10663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:31:23.188261 10676 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:31:23.189862 10676 webserver.cc:460] Webserver started at http://127.0.97.62:41531/ using document root <none> and password file <none>
I20260926 21:31:23.190685 10676 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:23.190840 10676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:23.191148 10676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:23.194307 10676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "13c5fd4b33d242be9494668253f4d75c"
format_stamp: "Formatted at 2026-09-26 21:31:23 on dist-test-slave-87bt"
server_key: "0cdefe6647ffe768c3c42a3b32790f26"
server_key_iv: "942447624557699695b47f9fce62e0c9"
server_key_version: "encryptionkey@0"
I20260926 21:31:23.195298 10676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "13c5fd4b33d242be9494668253f4d75c"
format_stamp: "Formatted at 2026-09-26 21:31:23 on dist-test-slave-87bt"
server_key: "0cdefe6647ffe768c3c42a3b32790f26"
server_key_iv: "942447624557699695b47f9fce62e0c9"
server_key_version: "encryptionkey@0"
I20260926 21:31:23.201088 10676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:31:23.204763 10691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:23.206187 10676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:31:23.206377 10676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "13c5fd4b33d242be9494668253f4d75c"
format_stamp: "Formatted at 2026-09-26 21:31:23 on dist-test-slave-87bt"
server_key: "0cdefe6647ffe768c3c42a3b32790f26"
server_key_iv: "942447624557699695b47f9fce62e0c9"
server_key_version: "encryptionkey@0"
I20260926 21:31:23.206645 10676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:23.247478 10676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:23.251885 10676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:23.252174 10676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:23.270136 10676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33067
I20260926 21:31:23.270219 10742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33067 every 8 connection(s)
I20260926 21:31:23.271489 10676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:31:23.275139 10743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:31:23.276906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 10676
I20260926 21:31:23.277136 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:31:23.277886 388 external_mini_cluster.cc:1428] Setting key 26f4d44c6dd5cd42e9ee00111853250c
I20260926 21:31:23.287492 10743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c: Bootstrap starting.
I20260926 21:31:23.291224 10743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c: Neither blocks nor log segments found. Creating new log.
I20260926 21:31:23.292397 10743 log.cc:824] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:31:23 dist-test-slave-87bt krb5kdc[10663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:31:23.295903 10743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c: No bootstrap required, opened a new log
I20260926 21:31:23.301931 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:23.279964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.0.97.62:33067)
0926 21:31:23.280788 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:23.280799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:23.280843 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:31:23.282524 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:23.282546 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:23.282919 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:23.283227 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:23.285029 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:23.286360 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:23.288126 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:23.288760 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:23.288895 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:23.293661 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:23.293669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:23.293708 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:23.293746 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:23.297551 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:23.298190 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:23.298198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:23.298206 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:23.298401 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:23.298776 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:23.298783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:23.298789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:23.299478 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:23.299769 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:23.300145 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 21:31:23.300553 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":397,"threads_started":1}
I20260926 21:31:23.305800 10743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c5fd4b33d242be9494668253f4d75c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } }
I20260926 21:31:23.306128 10743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:31:23.306208 10743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c5fd4b33d242be9494668253f4d75c, State: Initialized, Role: FOLLOWER
I20260926 21:31:23.306780 10743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c5fd4b33d242be9494668253f4d75c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } }
I20260926 21:31:23.306984 10743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:31:23.307080 10743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:31:23.307199 10743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:31:23.309216 10743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c5fd4b33d242be9494668253f4d75c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } }
I20260926 21:31:23.309787 10743 leader_election.cc:302] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13c5fd4b33d242be9494668253f4d75c; no voters:
I20260926 21:31:23.310575 10743 leader_election.cc:288] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:31:23.310662 10748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:31:23.311421 10748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [term 1 LEADER]: Becoming Leader. State: Replica: 13c5fd4b33d242be9494668253f4d75c, State: Running, Role: LEADER
I20260926 21:31:23.311991 10748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c5fd4b33d242be9494668253f4d75c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } }
I20260926 21:31:23.312844 10743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:31:23.314957 10749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13c5fd4b33d242be9494668253f4d75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c5fd4b33d242be9494668253f4d75c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } }
I20260926 21:31:23.314999 10750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c5fd4b33d242be9494668253f4d75c. Latest consensus state: current_term: 1 leader_uuid: "13c5fd4b33d242be9494668253f4d75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c5fd4b33d242be9494668253f4d75c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33067 } } }
I20260926 21:31:23.315327 10749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [sys.catalog]: This master's current role is: LEADER
I20260926 21:31:23.315411 10750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c [sys.catalog]: This master's current role is: LEADER
I20260926 21:31:23.323797 10754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:31:23.328809 10754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:31:23.337623 10754 catalog_manager.cc:1347] Generated new cluster ID: dcd93ab9f9e14a8c87cba05de78b4e4f
I20260926 21:31:23.337740 10754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:31:23.353928 10754 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:31:23.355558 10754 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:31:23.365207 10754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c: Generated new TSK 0
I20260926 21:31:23.366065 10754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:31:23.422976 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:31:23.751073 10769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:23.751345 10769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:23.751515 10769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:23.758682 10769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:23.758812 10769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:23.758989 10769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:31:23.768817 10769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:23.770145 10769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:23.771552 10769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:23.784258 10774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:23.784327 10775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:23.784409 10777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:23.787791 10769 server_base.cc:1034] running on GCE node
I20260926 21:31:23.788439 10769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:23.789399 10769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:23.790650 10769 hybrid_clock.cc:648] HybridClock initialized: now 1790458283790580 us; error 64 us; skew 500 ppm
I20260926 21:31:23.791014 10769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:31:23 dist-test-slave-87bt krb5kdc[10663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:31:23.794175 10769 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:31:23.795714 10769 webserver.cc:460] Webserver started at http://127.0.97.1:34147/ using document root <none> and password file <none>
I20260926 21:31:23.796449 10769 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:23.796590 10769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:23.796881 10769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:23.799809 10769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "ac201d40ad0b41838b4c95bedda011a7"
format_stamp: "Formatted at 2026-09-26 21:31:23 on dist-test-slave-87bt"
server_key: "cd79f245523e892e7a398916932f799e"
server_key_iv: "b206f6a3d469ed26f9f3e46428a16984"
server_key_version: "encryptionkey@0"
I20260926 21:31:23.800706 10769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ac201d40ad0b41838b4c95bedda011a7"
format_stamp: "Formatted at 2026-09-26 21:31:23 on dist-test-slave-87bt"
server_key: "cd79f245523e892e7a398916932f799e"
server_key_iv: "b206f6a3d469ed26f9f3e46428a16984"
server_key_version: "encryptionkey@0"
I20260926 21:31:23.806161 10769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:31:23.809448 10784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:23.810886 10769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 21:31:23.811056 10769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "ac201d40ad0b41838b4c95bedda011a7"
format_stamp: "Formatted at 2026-09-26 21:31:23 on dist-test-slave-87bt"
server_key: "cd79f245523e892e7a398916932f799e"
server_key_iv: "b206f6a3d469ed26f9f3e46428a16984"
server_key_version: "encryptionkey@0"
I20260926 21:31:23.811280 10769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:23.825026 10769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:23.829478 10769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:23.829762 10769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:23.830801 10769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:23.832072 10769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:23.832142 10769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:23.832298 10769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:23.832374 10769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:23.865049 10769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33259
I20260926 21:31:23.865070 10896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33259 every 8 connection(s)
I20260926 21:31:23.866467 10769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:31:23.873965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 10769
I20260926 21:31:23.874172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:31:23.874850 388 external_mini_cluster.cc:1428] Setting key e753d86f7814a3045013a33cb90553b4
Sep 26 21:31:23 dist-test-slave-87bt krb5kdc[10663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:31:23.887341 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:23.868897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60269 (local address 127.0.97.62:33067)
0926 21:31:23.869276 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:23.869280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:23.870294 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 21:31:23.872121 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:23.872127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:23.872216 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:23.872327 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:23.873921 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:23.874781 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:23.876914 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:23.877307 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:23.877399 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:23.881551 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:23.881556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:23.881559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:23.881589 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:23.884057 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:23.884772 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:23.884779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:23.884783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:23.884899 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:23.885346 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:23.885355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:23.885361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:23.885733 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:23.885885 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:23.886641 (+ 756us) server_negotiation.cc:299] Negotiation successful
0926 21:31:23.886880 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":190,"threads_started":1}
I20260926 21:31:23.888125 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:23.869481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.1:60269)
0926 21:31:23.870142 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:23.870185 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:23.871845 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:23.872546 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:23.872554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:23.873037 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:23.873719 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:23.873747 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:23.874937 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:23.874944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:23.876736 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:23.876749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:23.877537 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:23.877547 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:23.877873 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:23.878687 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:23.878723 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:23.881316 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:23.884212 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:31:23.884221 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:31:23.884238 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:31:23.884652 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:31:23.885014 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:31:23.885018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:31:23.885022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:31:23.885204 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:31:23.885864 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:23.885870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:23.886222 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 21:31:23.886539 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 21:31:23.886939 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":257,"threads_started":1}
I20260926 21:31:23.889653 10897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260926 21:31:23.889981 10897 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:23.890853 10897 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
I20260926 21:31:23.893224 10708 ts_manager.cc:194] Registered new tserver with Master: ac201d40ad0b41838b4c95bedda011a7 (127.0.97.1:33259)
I20260926 21:31:23.895484 10708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60269
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:31:23.927229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:31:24.251178 10903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:24.251464 10903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:24.251555 10903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:24.258741 10903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:24.258858 10903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:24.258975 10903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:31:24.268930 10903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:24.270588 10903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:24.272244 10903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:24.287149 10908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:24.287431 10911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:24.287720 10909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:24.288858 10903 server_base.cc:1034] running on GCE node
I20260926 21:31:24.289467 10903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:24.290403 10903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:24.291667 10903 hybrid_clock.cc:648] HybridClock initialized: now 1790458284291609 us; error 56 us; skew 500 ppm
I20260926 21:31:24.291991 10903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:31:24 dist-test-slave-87bt krb5kdc[10663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:31:24.295231 10903 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:31:24.296663 10903 webserver.cc:460] Webserver started at http://127.0.97.2:33605/ using document root <none> and password file <none>
I20260926 21:31:24.297397 10903 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:24.297533 10903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:24.297844 10903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:24.300832 10903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3"
format_stamp: "Formatted at 2026-09-26 21:31:24 on dist-test-slave-87bt"
server_key: "3d9dc2067ded0d0d62a69501e7210952"
server_key_iv: "ea2275f24b5c1ffcc7d7275f29c41336"
server_key_version: "encryptionkey@0"
I20260926 21:31:24.301592 10903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3"
format_stamp: "Formatted at 2026-09-26 21:31:24 on dist-test-slave-87bt"
server_key: "3d9dc2067ded0d0d62a69501e7210952"
server_key_iv: "ea2275f24b5c1ffcc7d7275f29c41336"
server_key_version: "encryptionkey@0"
I20260926 21:31:24.307129 10903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 21:31:24.310525 10918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:24.311856 10903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:31:24.312032 10903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3"
format_stamp: "Formatted at 2026-09-26 21:31:24 on dist-test-slave-87bt"
server_key: "3d9dc2067ded0d0d62a69501e7210952"
server_key_iv: "ea2275f24b5c1ffcc7d7275f29c41336"
server_key_version: "encryptionkey@0"
I20260926 21:31:24.312225 10903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:24.325966 10903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:24.330385 10903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:24.330709 10903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:24.331651 10903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:24.332964 10903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:24.333072 10903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:24.333211 10903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:24.333284 10903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:24.365522 10903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42855
I20260926 21:31:24.365638 11030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42855 every 8 connection(s)
I20260926 21:31:24.366812 10903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:31:24.369413 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 10903
I20260926 21:31:24.369622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:31:24.370018 388 external_mini_cluster.cc:1428] Setting key 17b7e82c57c72727488cbf2bcd0b2378
Sep 26 21:31:24 dist-test-slave-87bt krb5kdc[10663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:31:24.390494 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.369252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48647 (local address 127.0.97.62:33067)
0926 21:31:24.369460 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:24.369467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:24.371156 (+ 1689us) server_negotiation.cc:407] Connection header received
0926 21:31:24.373524 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:24.373528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:24.373589 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:24.373673 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:24.375698 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.376379 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.378816 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.379231 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.379326 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.383095 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:24.383104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:24.383112 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:24.383157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:24.386787 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:24.387647 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:24.387656 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:24.387662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:24.387815 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:24.388271 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:24.388280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:24.388286 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:24.388706 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:24.388856 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:24.389707 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 21:31:24.389981 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:31:24.391211 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.369842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.2:48647)
0926 21:31:24.370983 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:24.371059 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:24.373231 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:24.373940 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:24.373953 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:24.374750 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:24.375523 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.375550 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.376523 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.376531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.378674 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.378692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.379450 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.379460 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.379803 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.380565 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:24.380587 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:24.382806 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:24.386994 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:31:24.387002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:31:24.387018 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:31:24.387494 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:31:24.387930 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:31:24.387933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:31:24.387935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:31:24.388141 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:31:24.388963 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:24.388986 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:24.389329 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 21:31:24.389560 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 21:31:24.389957 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":667,"threads_started":1}
I20260926 21:31:24.392674 11031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260926 21:31:24.392997 11031 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:24.393711 11031 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:31:24.395239 10708 ts_manager.cc:194] Registered new tserver with Master: 1eb04d0cce4b4a878bf1d65cff9c77e3 (127.0.97.2:42855)
I20260926 21:31:24.396155 10708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:31:24.425462 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:31:24.757478 11036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:31:24.757771 11036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:31:24.757894 11036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:31:24.765256 11036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:31:24.765388 11036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:31:24.765587 11036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:31:24.775542 11036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:31:24.776919 11036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:31:24.778239 11036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:31:24.791558 11041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:24.791500 11044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:31:24.791518 11042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:31:24.792377 11036 server_base.cc:1034] running on GCE node
I20260926 21:31:24.792984 11036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:31:24.793936 11036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:31:24.795150 11036 hybrid_clock.cc:648] HybridClock initialized: now 1790458284795092 us; error 49 us; skew 500 ppm
I20260926 21:31:24.795459 11036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:31:24 dist-test-slave-87bt krb5kdc[10663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:31:24.798612 11036 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:31:24.800143 11036 webserver.cc:460] Webserver started at http://127.0.97.3:44787/ using document root <none> and password file <none>
I20260926 21:31:24.800828 11036 fs_manager.cc:359] Metadata directory not provided
I20260926 21:31:24.800974 11036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:31:24.801266 11036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:31:24.804267 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a65a4e58cbe4520a2304d4488edfe22"
format_stamp: "Formatted at 2026-09-26 21:31:24 on dist-test-slave-87bt"
server_key: "d797c61c609c60587e25203e97fc0e3b"
server_key_iv: "cdc63a519c6a6c50b09532e71cb39097"
server_key_version: "encryptionkey@0"
I20260926 21:31:24.805074 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a65a4e58cbe4520a2304d4488edfe22"
format_stamp: "Formatted at 2026-09-26 21:31:24 on dist-test-slave-87bt"
server_key: "d797c61c609c60587e25203e97fc0e3b"
server_key_iv: "cdc63a519c6a6c50b09532e71cb39097"
server_key_version: "encryptionkey@0"
I20260926 21:31:24.810232 11036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 21:31:24.813812 11051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:24.815260 11036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:31:24.815440 11036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "7a65a4e58cbe4520a2304d4488edfe22"
format_stamp: "Formatted at 2026-09-26 21:31:24 on dist-test-slave-87bt"
server_key: "d797c61c609c60587e25203e97fc0e3b"
server_key_iv: "cdc63a519c6a6c50b09532e71cb39097"
server_key_version: "encryptionkey@0"
I20260926 21:31:24.815665 11036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:31:24.828662 11036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:31:24.834039 11036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:31:24.834381 11036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:31:24.835634 11036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:31:24.837760 11036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:31:24.837857 11036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:24.837993 11036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:31:24.838065 11036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:31:24.874678 11036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41049
I20260926 21:31:24.874723 11163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41049 every 8 connection(s)
I20260926 21:31:24.875921 11036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:31:24.877471 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 11036
I20260926 21:31:24.877671 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:31:24.878073 388 external_mini_cluster.cc:1428] Setting key fdbdec364ab64a72540f0a14bdd62411
Sep 26 21:31:24 dist-test-slave-87bt krb5kdc[10663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:31:24.895457 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.878506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40871 (local address 127.0.97.62:33067)
0926 21:31:24.878711 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:24.878717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:24.880013 (+ 1296us) server_negotiation.cc:407] Connection header received
0926 21:31:24.882227 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:24.882231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:24.882293 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:24.882373 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:24.883797 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.884538 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.886030 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.886397 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.886544 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.889672 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:24.889676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:24.889680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:24.889711 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:24.892198 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:24.893223 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:24.893227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:24.893230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:24.893393 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:24.893919 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:24.893922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:24.893924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:24.894172 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:24.894280 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:24.895060 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 21:31:24.895210 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:31:24.896603 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.879058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33067 (local address 127.0.97.3:40871)
0926 21:31:24.879848 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:24.879911 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:24.881935 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:24.882605 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:24.882613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:24.883105 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:31:24.883655 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.883672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.884663 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.884666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.885915 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.885926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.886606 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.886611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.886794 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.887399 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:31:24.887419 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:31:24.889484 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:31:24.892369 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:31:24.892380 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:31:24.892397 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:31:24.893098 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:31:24.893508 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:31:24.893515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:31:24.893519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:31:24.893813 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:31:24.894280 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:31:24.894286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:31:24.894654 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 21:31:24.894981 (+ 327us) client_negotiation.cc:240] Negotiation successful
0926 21:31:24.895492 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":336,"threads_started":1}
I20260926 21:31:24.897856 11164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33067
I20260926 21:31:24.898169 11164 heartbeater.cc:464] Registering TS with master...
I20260926 21:31:24.898931 11164 heartbeater.cc:511] Master 127.0.97.62:33067 requested a full tablet report, sending...
I20260926 21:31:24.899269 10897 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
I20260926 21:31:24.900346 10707 ts_manager.cc:194] Registered new tserver with Master: 7a65a4e58cbe4520a2304d4488edfe22 (127.0.97.3:41049)
I20260926 21:31:24.901250 10707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40871
I20260926 21:31:24.905692 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:31:24.922396 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.909879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.0.97.62:33067)
0926 21:31:24.910086 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:24.910096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:24.910538 (+ 442us) server_negotiation.cc:407] Connection header received
0926 21:31:24.910695 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:24.910699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:24.910758 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:24.910835 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:24.912560 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.913309 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.914685 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.915152 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.915240 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.917550 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:24.917554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:24.917558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:24.917587 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:24.920093 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:24.920758 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:24.920761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:24.920764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:24.920901 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:24.921197 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:24.921200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:24.921202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:24.921478 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:24.921648 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:24.921922 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 21:31:24.922070 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:31:24.926270 10707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:31:24.933070 10707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:31:24.954213 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.944827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.0.97.1:33259)
0926 21:31:24.945509 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:24.945518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:24.945549 (+ 31us) server_negotiation.cc:407] Connection header received
0926 21:31:24.945768 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:24.945777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:24.946080 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:24.946281 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:31:24.947258 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.948719 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.951409 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.952884 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.953018 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.953263 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:24.953765 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 21:31:24.953909 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":365,"threads_started":1}
I20260926 21:31:24.954250 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.945802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41049 (local address 127.0.0.1:46640)
0926 21:31:24.946341 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:24.946363 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:24.946545 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:24.947729 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:24.947740 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:24.947770 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:31:24.948227 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.948239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.950033 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.950040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.951952 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.951963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.953262 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.953268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.953508 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.953542 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:31:24.953617 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 21:31:24.953832 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":95,"thread_start_us":240,"threads_started":1}
I20260926 21:31:24.954927 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.944526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33259 (local address 127.0.0.1:47320)
0926 21:31:24.945391 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:24.945445 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:24.945579 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:24.946605 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:24.946613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:24.946645 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:31:24.947037 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.947055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.948845 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.948851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.951199 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.951214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.953098 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.953110 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.953404 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.953463 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 21:31:24.953595 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 21:31:24.954532 (+ 937us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":735,"thread_start_us":446,"threads_started":1}
I20260926 21:31:24.954990 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.946054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.0.97.3:41049)
0926 21:31:24.947126 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:24.947136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:24.947168 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:31:24.947263 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:24.947270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:24.947516 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:24.948429 (+ 913us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:31:24.948701 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.949903 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.952102 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.953037 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.953392 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.953606 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:24.953673 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 21:31:24.953833 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":755,"threads_started":1}
I20260926 21:31:24.956418 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.946355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42855 (local address 127.0.0.1:38760)
0926 21:31:24.947762 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:24.947777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:24.947887 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:24.948714 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:24.948718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:24.948743 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:31:24.949165 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.949180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.951027 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.951033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.953170 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:24.953187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.955628 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.955635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:24.955809 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.955840 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:31:24.955952 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 21:31:24.956091 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":28,"thread_start_us":1151,"threads_started":1}
I20260926 21:31:24.956485 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:24.946829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.0.97.2:42855)
0926 21:31:24.947367 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:24.947375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:24.948008 (+ 633us) server_negotiation.cc:407] Connection header received
0926 21:31:24.948110 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:24.948119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:24.948328 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:24.948506 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:31:24.949335 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.950850 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.953392 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:24.955436 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:24.955574 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:24.955755 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:24.956011 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 21:31:24.956184 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260926 21:31:24.958688 11099 tablet_service.cc:1467] Processing CreateTablet for tablet 919693968b344f7eb7963f132166a127 (DEFAULT_TABLE table=test-table [id=f3bcf5e1a1f8450fad4638d352b4fc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:31:24.959326 10832 tablet_service.cc:1467] Processing CreateTablet for tablet 919693968b344f7eb7963f132166a127 (DEFAULT_TABLE table=test-table [id=f3bcf5e1a1f8450fad4638d352b4fc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:31:24.960188 11099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919693968b344f7eb7963f132166a127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:31:24.960681 10832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919693968b344f7eb7963f132166a127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:31:24.960244 10966 tablet_service.cc:1467] Processing CreateTablet for tablet 919693968b344f7eb7963f132166a127 (DEFAULT_TABLE table=test-table [id=f3bcf5e1a1f8450fad4638d352b4fc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:31:24.961494 10966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919693968b344f7eb7963f132166a127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:31:24.972391 11182 tablet_bootstrap.cc:492] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Bootstrap starting.
I20260926 21:31:24.975149 11183 tablet_bootstrap.cc:492] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Bootstrap starting.
I20260926 21:31:24.976025 11182 tablet_bootstrap.cc:654] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Neither blocks nor log segments found. Creating new log.
I20260926 21:31:24.977406 11182 log.cc:824] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Log is configured to *not* fsync() on all Append() calls
I20260926 21:31:24.978919 11183 tablet_bootstrap.cc:654] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Neither blocks nor log segments found. Creating new log.
I20260926 21:31:24.980526 11183 log.cc:824] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:31:24.981743 11184 tablet_bootstrap.cc:492] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Bootstrap starting.
I20260926 21:31:24.983856 11183 tablet_bootstrap.cc:492] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: No bootstrap required, opened a new log
I20260926 21:31:24.984139 11183 ts_tablet_manager.cc:1397] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 21:31:24.985008 11182 tablet_bootstrap.cc:492] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: No bootstrap required, opened a new log
I20260926 21:31:24.985324 11182 ts_tablet_manager.cc:1397] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260926 21:31:24.985390 11184 tablet_bootstrap.cc:654] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Neither blocks nor log segments found. Creating new log.
I20260926 21:31:24.986651 11184 log.cc:824] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Log is configured to *not* fsync() on all Append() calls
I20260926 21:31:24.990752 11184 tablet_bootstrap.cc:492] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: No bootstrap required, opened a new log
I20260926 21:31:24.990996 11184 ts_tablet_manager.cc:1397] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 21:31:24.992735 11183 raft_consensus.cc:348] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:24.993089 11183 raft_consensus.cc:374] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:31:24.993180 11183 raft_consensus.cc:729] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac201d40ad0b41838b4c95bedda011a7, State: Initialized, Role: FOLLOWER
I20260926 21:31:24.993773 11183 consensus_queue.cc:260] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:24.995533 11183 ts_tablet_manager.cc:1428] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 21:31:24.995998 11182 raft_consensus.cc:348] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:24.996447 11182 raft_consensus.cc:374] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:31:24.996549 11182 raft_consensus.cc:729] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a65a4e58cbe4520a2304d4488edfe22, State: Initialized, Role: FOLLOWER
I20260926 21:31:24.997166 11182 consensus_queue.cc:260] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:24.999692 11164 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
I20260926 21:31:25.000247 11182 ts_tablet_manager.cc:1428] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260926 21:31:25.001523 11184 raft_consensus.cc:348] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:25.001950 11184 raft_consensus.cc:374] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:31:25.002053 11184 raft_consensus.cc:729] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb04d0cce4b4a878bf1d65cff9c77e3, State: Initialized, Role: FOLLOWER
I20260926 21:31:25.002693 11184 consensus_queue.cc:260] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:25.003815 11031 heartbeater.cc:503] Master 127.0.97.62:33067 was elected leader, sending a full tablet report...
I20260926 21:31:25.004405 11184 ts_tablet_manager.cc:1428] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 21:31:25.079653 11188 raft_consensus.cc:482] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:31:25.079968 11188 raft_consensus.cc:504] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:25.081606 11188 leader_election.cc:288] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a65a4e58cbe4520a2304d4488edfe22 (127.0.97.3:41049), 1eb04d0cce4b4a878bf1d65cff9c77e3 (127.0.97.2:42855)
I20260926 21:31:25.088713 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:25.082143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39967 (local address 127.0.97.3:41049)
0926 21:31:25.082333 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:25.082341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:25.083156 (+ 815us) server_negotiation.cc:407] Connection header received
0926 21:31:25.083245 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:25.083250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:25.083328 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:25.083433 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:31:25.084044 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.084768 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.086570 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.087691 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.087796 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:25.087874 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:25.088308 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 21:31:25.088456 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:31:25.088846 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:25.082133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41049 (local address 127.0.97.1:39967)
0926 21:31:25.082898 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:25.082920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:25.083058 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:25.083571 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:25.083576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:25.083605 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:31:25.083889 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:25.083896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.084890 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.084898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:25.086372 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:25.086383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.087894 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.087900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:25.088083 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:25.088120 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 21:31:25.088225 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 21:31:25.088367 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":39,"thread_start_us":441,"threads_started":1}
I20260926 21:31:25.089391 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:25.083286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42983 (local address 127.0.97.2:42855)
0926 21:31:25.083472 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:25.083479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:25.083783 (+ 304us) server_negotiation.cc:407] Connection header received
0926 21:31:25.083971 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:25.083977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:25.084049 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:25.084144 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:31:25.084663 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.085339 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.087388 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.088466 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.088603 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:25.088691 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:25.089105 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 21:31:25.089226 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:31:25.089570 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:25.083051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42855 (local address 127.0.97.1:42983)
0926 21:31:25.083670 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 21:31:25.083686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:31:25.083830 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:31:25.084156 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:31:25.084161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:31:25.084182 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:31:25.084519 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:25.084527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.085474 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.085480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:25.087213 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:31:25.087227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.088711 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.088718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:31:25.088923 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:25.088954 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:31:25.089065 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 21:31:25.089210 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":2,"thread_start_us":338,"threads_started":1}
I20260926 21:31:25.089978 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919693968b344f7eb7963f132166a127" candidate_uuid: "ac201d40ad0b41838b4c95bedda011a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a65a4e58cbe4520a2304d4488edfe22" is_pre_election: true
I20260926 21:31:25.090626 11119 raft_consensus.cc:2454] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac201d40ad0b41838b4c95bedda011a7 in term 0.
I20260926 21:31:25.090560 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919693968b344f7eb7963f132166a127" candidate_uuid: "ac201d40ad0b41838b4c95bedda011a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" is_pre_election: true
I20260926 21:31:25.091089 10986 raft_consensus.cc:2454] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac201d40ad0b41838b4c95bedda011a7 in term 0.
I20260926 21:31:25.091346 10786 leader_election.cc:302] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a65a4e58cbe4520a2304d4488edfe22, ac201d40ad0b41838b4c95bedda011a7; no voters:
I20260926 21:31:25.091784 11188 raft_consensus.cc:2781] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:31:25.091969 11188 raft_consensus.cc:482] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:31:25.092089 11188 raft_consensus.cc:3037] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:31:25.094201 11188 raft_consensus.cc:504] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:25.094807 11188 leader_election.cc:288] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [CANDIDATE]: Term 1 election: Requested vote from peers 7a65a4e58cbe4520a2304d4488edfe22 (127.0.97.3:41049), 1eb04d0cce4b4a878bf1d65cff9c77e3 (127.0.97.2:42855)
I20260926 21:31:25.095293 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919693968b344f7eb7963f132166a127" candidate_uuid: "ac201d40ad0b41838b4c95bedda011a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a65a4e58cbe4520a2304d4488edfe22"
I20260926 21:31:25.095376 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919693968b344f7eb7963f132166a127" candidate_uuid: "ac201d40ad0b41838b4c95bedda011a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3"
I20260926 21:31:25.095485 11119 raft_consensus.cc:3037] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:31:25.095628 10986 raft_consensus.cc:3037] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:31:25.097736 11119 raft_consensus.cc:2454] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac201d40ad0b41838b4c95bedda011a7 in term 1.
I20260926 21:31:25.097798 10986 raft_consensus.cc:2454] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac201d40ad0b41838b4c95bedda011a7 in term 1.
I20260926 21:31:25.098160 10786 leader_election.cc:302] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a65a4e58cbe4520a2304d4488edfe22, ac201d40ad0b41838b4c95bedda011a7; no voters:
I20260926 21:31:25.098465 11188 raft_consensus.cc:2781] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:31:25.098914 11188 raft_consensus.cc:686] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 1 LEADER]: Becoming Leader. State: Replica: ac201d40ad0b41838b4c95bedda011a7, State: Running, Role: LEADER
I20260926 21:31:25.099447 11188 consensus_queue.cc:237] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } }
I20260926 21:31:25.103910 10708 catalog_manager.cc:5495] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac201d40ad0b41838b4c95bedda011a7 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ac201d40ad0b41838b4c95bedda011a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac201d40ad0b41838b4c95bedda011a7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 } health_report { overall_health: UNKNOWN } } }
W20260926 21:31:25.118736 10898 tablet.cc:2369] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:31:25.118752 11032 tablet.cc:2369] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:31:25.127463 11165 tablet.cc:2369] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:31:25.150063 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:25.144980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.97.1:33259)
0926 21:31:25.145170 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:25.145175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:25.145255 (+ 80us) server_negotiation.cc:407] Connection header received
0926 21:31:25.145454 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:25.145458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:25.145549 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:25.145669 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:31:25.146062 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.146886 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.148158 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.148564 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.148662 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:25.149073 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:31:25.149581 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:31:25.149687 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:25.149799 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 21:31:25.149864 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:31:25.158412 10986 raft_consensus.cc:1261] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 1 FOLLOWER]: Refusing update from remote peer ac201d40ad0b41838b4c95bedda011a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:31:25.158519 11119 raft_consensus.cc:1261] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 1 FOLLOWER]: Refusing update from remote peer ac201d40ad0b41838b4c95bedda011a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:31:25.159471 11188 consensus_queue.cc:1035] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb04d0cce4b4a878bf1d65cff9c77e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:31:25.159781 11193 consensus_queue.cc:1035] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a65a4e58cbe4520a2304d4488edfe22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:31:25.186488 10707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:31:25.186817 10707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:31:25.190543 10707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13c5fd4b33d242be9494668253f4d75c: Sending DeleteTablet for 3 replicas of tablet 919693968b344f7eb7963f132166a127
I20260926 21:31:25.191780 10832 tablet_service.cc:1514] Processing DeleteTablet for tablet 919693968b344f7eb7963f132166a127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:31:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47320
I20260926 21:31:25.192206 10966 tablet_service.cc:1514] Processing DeleteTablet for tablet 919693968b344f7eb7963f132166a127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:31:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38760
I20260926 21:31:25.192297 11099 tablet_service.cc:1514] Processing DeleteTablet for tablet 919693968b344f7eb7963f132166a127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:31:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46640
I20260926 21:31:25.194227 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 10769
I20260926 21:31:25.202687 11208 tablet_replica.cc:330] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: stopping tablet replica
I20260926 21:31:25.203310 11208 raft_consensus.cc:2229] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:31:25.203965 11208 raft_consensus.cc:2258] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:31:25.206481 11206 tablet_replica.cc:330] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: stopping tablet replica
I20260926 21:31:25.207026 11206 raft_consensus.cc:2229] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:31:25.207858 11206 raft_consensus.cc:2258] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:31:25.209096 11207 tablet_replica.cc:330] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: stopping tablet replica
I20260926 21:31:25.209503 11207 raft_consensus.cc:2229] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:31:25.210114 11207 raft_consensus.cc:2258] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:31:25.210732 11206 ts_tablet_manager.cc:1905] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:31:25.212024 11208 ts_tablet_manager.cc:1905] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:31:25.217600 11207 ts_tablet_manager.cc:1905] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:31:25.218647 11206 ts_tablet_manager.cc:1918] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:31:25.218827 11206 log.cc:1196] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/919693968b344f7eb7963f132166a127
I20260926 21:31:25.219496 11206 ts_tablet_manager.cc:1939] T 919693968b344f7eb7963f132166a127 P ac201d40ad0b41838b4c95bedda011a7: Deleting consensus metadata
I20260926 21:31:25.221771 10693 catalog_manager.cc:4841] TS ac201d40ad0b41838b4c95bedda011a7 (127.0.97.1:33259): tablet 919693968b344f7eb7963f132166a127 (table test-table [id=f3bcf5e1a1f8450fad4638d352b4fc75]) successfully deleted
I20260926 21:31:25.222965 11208 ts_tablet_manager.cc:1918] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:31:25.223135 11208 log.cc:1196] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/919693968b344f7eb7963f132166a127
I20260926 21:31:25.223829 11208 ts_tablet_manager.cc:1939] T 919693968b344f7eb7963f132166a127 P 7a65a4e58cbe4520a2304d4488edfe22: Deleting consensus metadata
Sep 26 21:31:25 dist-test-slave-87bt krb5kdc[10663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:31:25.225665 10693 catalog_manager.cc:4841] TS 7a65a4e58cbe4520a2304d4488edfe22 (127.0.97.3:41049): tablet 919693968b344f7eb7963f132166a127 (table test-table [id=f3bcf5e1a1f8450fad4638d352b4fc75]) successfully deleted
I20260926 21:31:25.227836 11207 ts_tablet_manager.cc:1918] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:31:25.228024 11207 log.cc:1196] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/919693968b344f7eb7963f132166a127
I20260926 21:31:25.228565 11207 ts_tablet_manager.cc:1939] T 919693968b344f7eb7963f132166a127 P 1eb04d0cce4b4a878bf1d65cff9c77e3: Deleting consensus metadata
I20260926 21:31:25.230413 10693 catalog_manager.cc:4841] TS 1eb04d0cce4b4a878bf1d65cff9c77e3 (127.0.97.2:42855): tablet 919693968b344f7eb7963f132166a127 (table test-table [id=f3bcf5e1a1f8450fad4638d352b4fc75]) successfully deleted
I20260926 21:31:25.232750 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:25.198136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.0.97.1:33259)
0926 21:31:25.198371 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:25.198379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:25.212695 (+ 14316us) server_negotiation.cc:407] Connection header received
0926 21:31:25.213159 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:25.213167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:25.213254 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:25.213328 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:25.215267 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.216633 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.218747 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.219478 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.219598 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:25.226368 (+ 6770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:25.226404 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:25.226426 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:25.226511 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:25.229796 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:25.230710 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:25.230720 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:25.230723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:25.230882 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:25.231252 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:25.231257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:25.231261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:25.231614 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:25.231897 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:25.232185 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 21:31:25.232414 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:31:25.233536 10892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:25.331214 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 10769
I20260926 21:31:25.350205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 10903
Sep 26 21:31:25 dist-test-slave-87bt krb5kdc[10663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:31:25.364879 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:25.351085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.0.97.2:42855)
0926 21:31:25.351276 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:25.351282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:25.351304 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:31:25.351378 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:25.351384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:25.351464 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:25.351569 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:25.352804 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.353540 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.354871 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.355280 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.355414 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:25.359323 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:25.359339 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:25.359355 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:25.359398 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:25.362247 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:25.363033 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:25.363041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:25.363043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:25.363162 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:25.363531 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:25.363534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:25.363536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:25.363886 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:25.364099 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:25.364340 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 21:31:25.364520 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:31:25.365708 11026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:25.457731 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 10903
I20260926 21:31:25.477090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 11036
Sep 26 21:31:25 dist-test-slave-87bt krb5kdc[10663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:31:25.492053 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:31:25.477853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.0.97.3:41049)
0926 21:31:25.478064 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 21:31:25.478070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:31:25.478091 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:31:25.478278 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:31:25.478284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:31:25.478357 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:31:25.478510 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:31:25.479948 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.480698 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.481927 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:31:25.482330 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:31:25.482493 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:31:25.486881 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:31:25.486898 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:31:25.486914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:31:25.486949 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:31:25.489546 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:25.490222 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:25.490228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:25.490231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:25.490350 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:31:25.490842 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:31:25.490845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:31:25.490847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:31:25.491176 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:31:25.491396 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 21:31:25.491633 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 21:31:25.491797 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:31:25.492677 11159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:25.581782 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 11036
I20260926 21:31:25.598673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 10676
I20260926 21:31:25.599774 10738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:31:25.706585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 10676
2026-09-26T21:31:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4230 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:31:25 dist-test-slave-87bt krb5kdc[11216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:31:25 dist-test-slave-87bt krb5kdc[11216](info): set up 2 sockets
Sep 26 21:31:25 dist-test-slave-87bt krb5kdc[11216](info): commencing operation
krb5kdc: starting...
W20260926 21:31:26.811299 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:31:26 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:31:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 21:31:27.146761 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUh_ENg/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/postgres -l logfile start
2026-09-26 21:31:29.673 UTC [11246] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 21:31:29.673 UTC [11246] LOG: listening on IPv4 address "127.0.97.20", port 47151
2026-09-26 21:31:29.677 UTC [11246] LOG: listening on Unix socket "/tmp/.s.PGSQL.47151"
2026-09-26 21:31:29.697 UTC [11249] LOG: database system was shut down at 2026-09-26 21:31:28 UTC
2026-09-26 21:31:29.701 UTC [11246] LOG: database system is ready to accept connections
I20260926 21:31:30.675135 388 mini_postgres.cc:96] Postgres bound to 47151
2026-09-26 21:31:30.692 UTC [11257] FATAL: database "slave" does not exist
127.0.97.20:47151 - accepting connections
I20260926 21:31:30.693096 388 mini_ranger.cc:161] Starting Ranger...
I20260926 21:31:30.719661 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 21:31:31.203379 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 21:31:31.203491 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-admin
2026-09-26 21:31:31,488 [I] DB FLAVOR :POSTGRES
2026-09-26 21:31:31,488 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 21:31:31,488 [I] Checking connection..
2026-09-26 21:31:31,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 21:31:31,737 [I] Checking connection passed.
2026-09-26 21:31:31,738 [I] --------- Verifying version history table ---------
2026-09-26 21:31:31,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:31:32,005 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 21:31:32,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:31:32,241 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 21:31:32,241 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 21:31:32,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 21:31:32.498 UTC [11321] WARNING: there is no transaction in progress
2026-09-26 21:31:32,504 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 21:31:32,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:31:32,737 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 21:31:32,737 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 21:31:32,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 21:31:32,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 21:31:33,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-87bt.c.gcp-upstream.internal','N') ;"
2026-09-26 21:31:33,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 21:31:33,722 [I] Table x_portal_user does not exist in database ranger
2026-09-26 21:31:33,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 21:31:33,978 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 21:31:33,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 21:31:34,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 21:31:34,481 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 21:31:34,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 21:31:35.469 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.485 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.501 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.517 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.533 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.554 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.610 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.617 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.631 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.642 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.652 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.666 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.676 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.683 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.690 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:35.697 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:31:36,328 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 21:31:36,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 21:31:36,557 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 21:31:36,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 21:31:36,810 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 21:31:36,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 21:31:37,061 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 21:31:37,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 21:31:37,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 21:31:37,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-87bt.c.gcp-upstream.internal';"
2026-09-26 21:31:37,789 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 21:31:37,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 21:31:38,028 [I] DB_PATCHES have already been applied
I20260926 21:31:38.034415 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUh_ENg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 21:31:38.034579 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 21:31:38.049743 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:45743
Sep 26, 2026 9:31:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:31:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:31:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:31:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:31:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:31:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:31:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 21:31:39.205862 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.513s user 0.003s sys 0.036s
Sep 26, 2026 9:32:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:32:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 9:32:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:32:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:32:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 21:32:01.792021 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 21:32:01 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458321, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 21:32:01.986483 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 21:32:02.009769 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 21:32:03.007236 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 21:32:03.007403 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms
I20260926 21:32:03.007515 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:47151
2026-09-26 21:32:03,429 [I] DB FLAVOR :POSTGRES
2026-09-26 21:32:03,429 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 21:32:03,430 [I] Checking connection
2026-09-26 21:32:03,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 21:32:03,677 [I] connection success
2026-09-26 21:32:03,678 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 21:32:03,678 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 21:32:03,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 21:32:03,914 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 21:32:03,915 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 21:32:03,915 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 21:32:03,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:47151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 21:32:04,179 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 21:32:04.736209 388 mini_ranger_kms.cc:310] Created kms service
I20260926 21:32:04.923684 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 21:32:04.973448 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 21:32:04 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458321, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:32:05.464339 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 21:32:05.464658 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 21:32:05.464712 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 21:32:05.484030 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11610
I20260926 21:32:05.484247 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:49811
Sep 26, 2026 9:32:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:32:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:32:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:32:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:32:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:32:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 21:32:07 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458327, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:32:07.845590 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.859s user 0.115s sys 0.330s
I20260926 21:32:07.845849 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 21:32:07.846071 388 mini_ranger_kms.cc:401] 127.0.97.20:49811/kms/v1/keys
Sep 26 21:32:08 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458327, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:32:08 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458327, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:32:10 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:32:10.927160 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39971
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39971
--ranger_config_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49811/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:32:11.281270 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:32:11.281669 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:32:11.281845 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:32:11.289187 11660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:32:11.289361 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:32:11.289474 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:32:11.289579 11660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:32:11.289671 11660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:32:11.299871 11660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:49811/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39971
--ranger_config_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:32:11.301234 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:32:11.302659 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:32:11.316143 11665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:32:11.316143 11666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:32:11.316174 11668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:32:11.316449 11660 server_base.cc:1034] running on GCE node
I20260926 21:32:11.317672 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:32:11.319257 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:32:11.320470 11660 hybrid_clock.cc:648] HybridClock initialized: now 1790458331320429 us; error 35 us; skew 500 ppm
I20260926 21:32:11.320847 11660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:32:11 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:32:11.328553 11660 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:32:11.330061 11660 webserver.cc:460] Webserver started at http://127.0.97.62:42657/ using document root <none> and password file <none>
I20260926 21:32:11.330821 11660 fs_manager.cc:359] Metadata directory not provided
I20260926 21:32:11.330960 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:32:11.331272 11660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:32:11 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:32:11.394923 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "745f5db686434b23840f08ecb8b61da1"
format_stamp: "Formatted at 2026-09-26 21:32:11 on dist-test-slave-87bt"
server_key: "4bd89ce9184ae1378a06bad4be4d5a18"
server_key_iv: "f560f7e6acb29df5771b095a0db438f2"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:11.395843 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "745f5db686434b23840f08ecb8b61da1"
format_stamp: "Formatted at 2026-09-26 21:32:11 on dist-test-slave-87bt"
server_key: "4bd89ce9184ae1378a06bad4be4d5a18"
server_key_iv: "f560f7e6acb29df5771b095a0db438f2"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:11.402123 11660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 21:32:11.423488 11675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:32:11.425092 11660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:32:11.425278 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "745f5db686434b23840f08ecb8b61da1"
format_stamp: "Formatted at 2026-09-26 21:32:11 on dist-test-slave-87bt"
server_key: "4bd89ce9184ae1378a06bad4be4d5a18"
server_key_iv: "f560f7e6acb29df5771b095a0db438f2"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:11.425503 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:32:11.444816 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:32:11.449193 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:32:11.449505 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:32:11.470705 11660 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39971
I20260926 21:32:11.470727 11726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39971 every 8 connection(s)
I20260926 21:32:11.472123 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:32:11.475863 11727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:32:11.477648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 11660
I20260926 21:32:11.477842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:32:11.478540 388 external_mini_cluster.cc:1428] Setting key 61f2b6c33260cb1da02c90fe94677032
I20260926 21:32:11.488520 11727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1: Bootstrap starting.
I20260926 21:32:11.492795 11727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1: Neither blocks nor log segments found. Creating new log.
Sep 26 21:32:11 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:32:11.494582 11727 log.cc:824] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1: Log is configured to *not* fsync() on all Append() calls
I20260926 21:32:11.497706 11727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1: No bootstrap required, opened a new log
I20260926 21:32:11.503490 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:11.481222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.0.97.62:39971)
0926 21:32:11.482066 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:11.482079 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:11.482121 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:32:11.483849 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:11.483874 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:11.484302 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:11.484673 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:32:11.486290 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:11.487759 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:11.489496 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:11.490215 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:11.490372 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:11.495523 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:32:11.495533 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:32:11.495551 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:32:11.495596 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:32:11.498851 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:11.499661 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:11.499669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:11.499677 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:11.499841 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:11.500270 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:11.500277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:11.500280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:11.501050 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:32:11.501411 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:11.501859 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 21:32:11.502260 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":384,"threads_started":1}
I20260926 21:32:11.506379 11727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f5db686434b23840f08ecb8b61da1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } }
I20260926 21:32:11.506870 11727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:32:11.506981 11727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745f5db686434b23840f08ecb8b61da1, State: Initialized, Role: FOLLOWER
I20260926 21:32:11.507642 11727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f5db686434b23840f08ecb8b61da1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } }
I20260926 21:32:11.507925 11727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:32:11.508093 11727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:32:11.508273 11727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:32:11.511165 11727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f5db686434b23840f08ecb8b61da1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } }
I20260926 21:32:11.511763 11727 leader_election.cc:302] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 745f5db686434b23840f08ecb8b61da1; no voters:
I20260926 21:32:11.512419 11727 leader_election.cc:288] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:32:11.512650 11732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:32:11.513540 11732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [term 1 LEADER]: Becoming Leader. State: Replica: 745f5db686434b23840f08ecb8b61da1, State: Running, Role: LEADER
I20260926 21:32:11.514153 11732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f5db686434b23840f08ecb8b61da1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } }
I20260926 21:32:11.514974 11727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:32:11.517746 11733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "745f5db686434b23840f08ecb8b61da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f5db686434b23840f08ecb8b61da1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } } }
I20260926 21:32:11.517721 11734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745f5db686434b23840f08ecb8b61da1. Latest consensus state: current_term: 1 leader_uuid: "745f5db686434b23840f08ecb8b61da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f5db686434b23840f08ecb8b61da1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39971 } } }
I20260926 21:32:11.518386 11733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [sys.catalog]: This master's current role is: LEADER
I20260926 21:32:11.518412 11734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1 [sys.catalog]: This master's current role is: LEADER
I20260926 21:32:11.525427 11737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:32:11.525774 11727 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 21:32:11.538709 11737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:32:11.550882 11737 catalog_manager.cc:1347] Generated new cluster ID: 6212e941b560456a9b4cc792f01a20a2
I20260926 21:32:11.551011 11737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:32:11.571491 11737 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:32:11.573487 11737 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:32:11.586491 11737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1: Generated new TSK 0
I20260926 21:32:11.587673 11737 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:32:12.348445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39971
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:32:12 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 21:32:12.807063 11772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:32:12.807497 11772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:32:12.807691 11772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:32:12.816841 11772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:32:12.817090 11772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:32:12.817332 11772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:32:12.830571 11772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:49811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:32:12.832897 11772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:32:12.834869 11772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:32:12.853816 11782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:32:12.862707 11783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:32:12.863430 11772 server_base.cc:1034] running on GCE node
W20260926 21:32:12.864411 11785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:32:12.865370 11772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:32:12.866674 11772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:32:12.870136 11772 hybrid_clock.cc:648] HybridClock initialized: now 1790458332870055 us; error 70 us; skew 500 ppm
I20260926 21:32:12.870671 11772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:32:12 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:32:12.875253 11772 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:32:12.877987 11772 webserver.cc:460] Webserver started at http://127.0.97.1:38791/ using document root <none> and password file <none>
I20260926 21:32:12.879084 11772 fs_manager.cc:359] Metadata directory not provided
I20260926 21:32:12.879249 11772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:32:12.879673 11772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:32:12 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:32:12.909578 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "5627ab1f5d474fa599335f6ceaf9ae49"
format_stamp: "Formatted at 2026-09-26 21:32:12 on dist-test-slave-87bt"
server_key: "5b3bbf260206bc83542cf75cc720eb1e"
server_key_iv: "ee65d2c9942a37a90f80d6e4eb549094"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:12.915216 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5627ab1f5d474fa599335f6ceaf9ae49"
format_stamp: "Formatted at 2026-09-26 21:32:12 on dist-test-slave-87bt"
server_key: "5b3bbf260206bc83542cf75cc720eb1e"
server_key_iv: "ee65d2c9942a37a90f80d6e4eb549094"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:12.928635 11772 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.010s sys 0.001s
I20260926 21:32:12.947250 11792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:32:12.949091 11772 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 21:32:12.949318 11772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "5627ab1f5d474fa599335f6ceaf9ae49"
format_stamp: "Formatted at 2026-09-26 21:32:12 on dist-test-slave-87bt"
server_key: "5b3bbf260206bc83542cf75cc720eb1e"
server_key_iv: "ee65d2c9942a37a90f80d6e4eb549094"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:12.949563 11772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:32:12.966655 11772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:32:12.972285 11772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:32:12.972716 11772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:32:12.994233 11772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:32:12.999271 11772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:32:12.999487 11772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 21:32:12.999723 11772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:32:12.999886 11772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:32:13 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:32:13.355719 11772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41813
I20260926 21:32:13.355999 11904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41813 every 8 connection(s)
I20260926 21:32:13.357180 11772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:32:13.362926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 11772
I20260926 21:32:13.363124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:32:13.363952 388 external_mini_cluster.cc:1428] Setting key 7111950c282c96a97e06dd76ed0ac134
Sep 26 21:32:13 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:32:13.406024 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:13.365150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51517 (local address 127.0.97.62:39971)
0926 21:32:13.381373 (+ 16223us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:13.381381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:13.384665 (+ 3284us) server_negotiation.cc:407] Connection header received
0926 21:32:13.386183 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:13.386190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:13.386298 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:13.386417 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:32:13.388671 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:13.389779 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:13.392250 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:13.392898 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:13.393005 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:13.397831 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:32:13.397839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:32:13.397844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:32:13.397886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:32:13.401464 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:13.402549 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:13.402555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:13.402559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:13.402738 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:13.403310 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:13.403316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:13.403319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:13.403687 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:32:13.403822 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:13.405002 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0926 21:32:13.405242 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16076,"thread_start_us":15919,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:32:13.406947 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:13.362083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39971 (local address 127.0.97.1:51517)
0926 21:32:13.383083 (+ 21000us) negotiation.cc:107] Waiting for socket to connect
0926 21:32:13.383164 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 21:32:13.385762 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:32:13.386924 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:32:13.386939 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:32:13.387572 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:32:13.388429 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:13.388460 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:13.390006 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:13.390011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:13.392054 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:13.392071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:13.393391 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:13.393399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:13.393746 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:13.394668 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:32:13.394701 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:32:13.397529 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:32:13.401698 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:32:13.401708 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:32:13.401729 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:32:13.402364 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:32:13.402904 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:32:13.402910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:32:13.402915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:32:13.403174 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:32:13.403946 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:32:13.403955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:32:13.404504 (+ 549us) client_negotiation.cc:769] Sending connection context
0926 21:32:13.404830 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 21:32:13.405431 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20639,"thread_start_us":20411,"threads_started":1}
I20260926 21:32:13.410534 11905 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39971
I20260926 21:32:13.410872 11905 heartbeater.cc:464] Registering TS with master...
I20260926 21:32:13.411761 11905 heartbeater.cc:511] Master 127.0.97.62:39971 requested a full tablet report, sending...
I20260926 21:32:13.414287 11692 ts_manager.cc:194] Registered new tserver with Master: 5627ab1f5d474fa599335f6ceaf9ae49 (127.0.97.1:41813)
I20260926 21:32:13.416399 11692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:32:13.442701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39971
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:32:13 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 21:32:13.932709 11911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:32:13.933059 11911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:32:13.933213 11911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:32:13.941428 11911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:32:13.941601 11911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:32:13.941788 11911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:32:13.952323 11911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:49811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:32:13.953833 11911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:32:13.955350 11911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:32:13.969385 11926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:32:13.969552 11925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:32:13.969620 11928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:32:13.971169 11911 server_base.cc:1034] running on GCE node
I20260926 21:32:13.971791 11911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:32:13.972858 11911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:32:13.974088 11911 hybrid_clock.cc:648] HybridClock initialized: now 1790458333974033 us; error 49 us; skew 500 ppm
I20260926 21:32:13.974499 11911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:32:13 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:32:13.978375 11911 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:32:13.980146 11911 webserver.cc:460] Webserver started at http://127.0.97.2:36787/ using document root <none> and password file <none>
I20260926 21:32:13.981034 11911 fs_manager.cc:359] Metadata directory not provided
I20260926 21:32:13.981163 11911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:32:13.981499 11911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:32:13 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:32:13.999042 11911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "0e1c42c82cfe476086a5c81003e31b78"
format_stamp: "Formatted at 2026-09-26 21:32:13 on dist-test-slave-87bt"
server_key: "0c17ab86bddbe29f26b93e6009cf8e86"
server_key_iv: "ea26409243a9fdea87267c8c5ade503b"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:14.000622 11911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0e1c42c82cfe476086a5c81003e31b78"
format_stamp: "Formatted at 2026-09-26 21:32:13 on dist-test-slave-87bt"
server_key: "0c17ab86bddbe29f26b93e6009cf8e86"
server_key_iv: "ea26409243a9fdea87267c8c5ade503b"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:14.007805 11911 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260926 21:32:14.028013 11935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:32:14.029819 11911 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:32:14.030058 11911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "0e1c42c82cfe476086a5c81003e31b78"
format_stamp: "Formatted at 2026-09-26 21:32:13 on dist-test-slave-87bt"
server_key: "0c17ab86bddbe29f26b93e6009cf8e86"
server_key_iv: "ea26409243a9fdea87267c8c5ade503b"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:14.030304 11911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:32:14.050201 11911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:32:14.055086 11911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:32:14.055404 11911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:32:14.056396 11911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:32:14.057901 11911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:32:14.058003 11911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:32:14.058563 11911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:32:14.058660 11911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260926 21:32:14.096642 11911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39883
I20260926 21:32:14.096697 12047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39883 every 8 connection(s)
I20260926 21:32:14.097980 11911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:32:14.102126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 11911
I20260926 21:32:14.102355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:32:14.103031 388 external_mini_cluster.cc:1428] Setting key 263d81ac97f1c8b50c93144a23e5a4ac
Sep 26 21:32:14 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:32:14.123669 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.100716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57853 (local address 127.0.97.62:39971)
0926 21:32:14.101255 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:14.101261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:14.105716 (+ 4455us) server_negotiation.cc:407] Connection header received
0926 21:32:14.105779 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:14.105784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:14.105879 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:14.105988 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:32:14.108080 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.109142 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.111838 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.112943 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.113096 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.117154 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:32:14.117161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:32:14.117166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:32:14.117207 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:32:14.120238 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:14.121052 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:14.121057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:14.121063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:14.121282 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:14.121768 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:14.121772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:14.121776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:14.122057 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:32:14.122230 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:14.123037 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 21:32:14.123287 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":281,"threads_started":1}
I20260926 21:32:14.124811 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.101262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39971 (local address 127.0.97.2:57853)
0926 21:32:14.102996 (+ 1734us) negotiation.cc:107] Waiting for socket to connect
0926 21:32:14.103044 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:32:14.105315 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:32:14.106288 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:32:14.106304 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:32:14.106996 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:32:14.107863 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.107901 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.109332 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.109341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.111627 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.111657 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.113295 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.113303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.113616 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.114354 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:32:14.114378 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:32:14.116884 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:32:14.120398 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:32:14.120408 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:32:14.120429 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:32:14.120906 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:32:14.121405 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:32:14.121411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:32:14.121414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:32:14.121633 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:32:14.122201 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:32:14.122212 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:32:14.122660 (+ 448us) client_negotiation.cc:769] Sending connection context
0926 21:32:14.122939 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 21:32:14.123491 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":1272,"threads_started":1}
I20260926 21:32:14.126504 12048 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39971
I20260926 21:32:14.126843 12048 heartbeater.cc:464] Registering TS with master...
I20260926 21:32:14.127601 12048 heartbeater.cc:511] Master 127.0.97.62:39971 requested a full tablet report, sending...
I20260926 21:32:14.129169 11692 ts_manager.cc:194] Registered new tserver with Master: 0e1c42c82cfe476086a5c81003e31b78 (127.0.97.2:39883)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:32:14.130155 11692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:32:14.165331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39971
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:32:14.420141 11905 heartbeater.cc:503] Master 127.0.97.62:39971 was elected leader, sending a full tablet report...
W20260926 21:32:14.526597 12054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:32:14.526944 12054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:32:14.527129 12054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:32:14.534968 12054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:32:14.535079 12054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:32:14.535292 12054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:32:14.545012 12054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:49811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:32:14.546299 12054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:32:14.547629 12054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:32:14.561477 12062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:32:14.561609 12059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:32:14.561576 12060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:32:14.563169 12054 server_base.cc:1034] running on GCE node
I20260926 21:32:14.563800 12054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:32:14.564818 12054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:32:14.566021 12054 hybrid_clock.cc:648] HybridClock initialized: now 1790458334565968 us; error 45 us; skew 500 ppm
I20260926 21:32:14.566344 12054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:32:14 dist-test-slave-87bt krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:32:14.569995 12054 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:32:14.571547 12054 webserver.cc:460] Webserver started at http://127.0.97.3:45111/ using document root <none> and password file <none>
I20260926 21:32:14.572217 12054 fs_manager.cc:359] Metadata directory not provided
I20260926 21:32:14.572317 12054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:32:14.572582 12054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:32:14 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:32:14.587394 12054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "fb3e29d3d6894d519c0dccb0c9e34e30"
format_stamp: "Formatted at 2026-09-26 21:32:14 on dist-test-slave-87bt"
server_key: "84649221ad77698abb6e6b4e291c684c"
server_key_iv: "05f81c6bd44294be2db06466c5e78a7a"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:14.588339 12054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fb3e29d3d6894d519c0dccb0c9e34e30"
format_stamp: "Formatted at 2026-09-26 21:32:14 on dist-test-slave-87bt"
server_key: "84649221ad77698abb6e6b4e291c684c"
server_key_iv: "05f81c6bd44294be2db06466c5e78a7a"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:14.594024 12054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 21:32:14.611006 12069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:32:14.613106 12054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 21:32:14.613315 12054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "fb3e29d3d6894d519c0dccb0c9e34e30"
format_stamp: "Formatted at 2026-09-26 21:32:14 on dist-test-slave-87bt"
server_key: "84649221ad77698abb6e6b4e291c684c"
server_key_iv: "05f81c6bd44294be2db06466c5e78a7a"
server_key_version: "kuduclusterkey@0"
I20260926 21:32:14.613584 12054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:32:14.645408 12054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:32:14.649932 12054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:32:14.650285 12054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:32:14.651279 12054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:32:14.652663 12054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:32:14.652740 12054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:32:14.652872 12054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:32:14.652923 12054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:32:14.689720 12054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36957
I20260926 21:32:14.689759 12181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36957 every 8 connection(s)
I20260926 21:32:14.691082 12054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:32:14.692713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 12054
I20260926 21:32:14.692896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:32:14.693387 388 external_mini_cluster.cc:1428] Setting key ae4eb80b875d43a09144416403364266
Sep 26 21:32:14 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:32:14.712159 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.693856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49431 (local address 127.0.97.62:39971)
0926 21:32:14.694335 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:14.694339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:14.695606 (+ 1267us) server_negotiation.cc:407] Connection header received
0926 21:32:14.697700 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:14.697705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:14.697790 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:14.697879 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:32:14.699361 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.700048 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.701518 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.701932 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.702021 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.705234 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:32:14.705239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:32:14.705242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:32:14.705275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:32:14.708791 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:14.710008 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:14.710012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:14.710015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:14.710144 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:14.710632 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:14.710636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:14.710638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:14.710910 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:32:14.711034 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:14.711719 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 21:32:14.711893 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":235,"threads_started":1}
I20260926 21:32:14.713168 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.694547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39971 (local address 127.0.97.3:49431)
0926 21:32:14.695406 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 21:32:14.695484 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 21:32:14.697443 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:32:14.698132 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:32:14.698141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:32:14.698637 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:32:14.699222 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.699239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.700170 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.700173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.701398 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.701410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.702112 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.702116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.702302 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.702998 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:32:14.703020 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:32:14.705009 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:32:14.709109 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:32:14.709116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:32:14.709132 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:32:14.709877 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:32:14.710240 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:32:14.710244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:32:14.710246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:32:14.710525 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:32:14.711028 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:32:14.711034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:32:14.711370 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 21:32:14.711626 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 21:32:14.712025 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":348,"threads_started":1}
I20260926 21:32:14.714635 12182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39971
I20260926 21:32:14.714974 12182 heartbeater.cc:464] Registering TS with master...
I20260926 21:32:14.715806 12182 heartbeater.cc:511] Master 127.0.97.62:39971 requested a full tablet report, sending...
I20260926 21:32:14.717576 11692 ts_manager.cc:194] Registered new tserver with Master: fb3e29d3d6894d519c0dccb0c9e34e30 (127.0.97.3:36957)
I20260926 21:32:14.718510 11692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49431
I20260926 21:32:14.722668 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:32:14.741997 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.727509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.0.97.62:39971)
0926 21:32:14.727685 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:14.727690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:14.728140 (+ 450us) server_negotiation.cc:407] Connection header received
0926 21:32:14.728282 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:14.728289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:14.728374 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:14.728516 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:32:14.729957 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.731001 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.732337 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.732858 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.732978 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.735560 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:32:14.735565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:32:14.735569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:32:14.735607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:32:14.739120 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:14.739982 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:14.739987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:14.739989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:14.740155 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:14.740626 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:14.740630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:14.740632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:14.740963 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:32:14.741181 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:14.741559 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 21:32:14.741754 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:32:14.746672 11692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:32:14.754079 11692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:32:14.782049 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.768280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36957 (local address 127.0.0.1:33110)
0926 21:32:14.769186 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 21:32:14.769239 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 21:32:14.769401 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:32:14.770619 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:32:14.770631 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:32:14.770675 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:32:14.771190 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.771207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.773838 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.773847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.777438 (+ 3591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.777463 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.780626 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.780636 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.780935 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.781026 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 21:32:14.781201 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 21:32:14.781497 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":587,"threads_started":1}
I20260926 21:32:14.783354 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.768388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.0.97.3:36957)
0926 21:32:14.769518 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:14.769527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:14.769563 (+ 36us) server_negotiation.cc:407] Connection header received
0926 21:32:14.769665 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:14.769673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:14.769973 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:14.770209 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:32:14.771494 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.773617 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.777749 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.779917 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.780095 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.780427 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:14.782602 (+ 2175us) server_negotiation.cc:299] Negotiation successful
0926 21:32:14.782882 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":811,"threads_started":1}
I20260926 21:32:14.795321 12117 tablet_service.cc:1467] Processing CreateTablet for tablet 22908c80194e42408d17675228e31811 (DEFAULT_TABLE table=test-table [id=0f4da8d2263841b6914895457b519c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:32:14.797127 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22908c80194e42408d17675228e31811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:32:14.797940 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.768740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39883 (local address 127.0.0.1:37738)
0926 21:32:14.783972 (+ 15232us) negotiation.cc:107] Waiting for socket to connect
0926 21:32:14.783999 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 21:32:14.784175 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:32:14.787101 (+ 2926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:32:14.787109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:32:14.787149 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:32:14.787418 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.787434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.791292 (+ 3858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.791298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.793265 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.793283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.797107 (+ 3824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.797114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.797319 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.797403 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 21:32:14.797477 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 21:32:14.797633 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15043,"thread_start_us":28000,"threads_started":1}
I20260926 21:32:14.799036 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.769760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.0.97.2:39883)
0926 21:32:14.786173 (+ 16413us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:14.786185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:14.786228 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:32:14.786379 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:14.786391 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:14.786803 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:14.787670 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:32:14.787977 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.791104 (+ 3127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.793514 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.794934 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.798193 (+ 3259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.798466 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:14.798543 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 21:32:14.798717 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16213,"thread_start_us":16074,"threads_started":1}
I20260926 21:32:14.803859 11983 tablet_service.cc:1467] Processing CreateTablet for tablet 22908c80194e42408d17675228e31811 (DEFAULT_TABLE table=test-table [id=0f4da8d2263841b6914895457b519c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:32:14.804857 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.769695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41813 (local address 127.0.0.1:35452)
0926 21:32:14.790906 (+ 21211us) negotiation.cc:107] Waiting for socket to connect
0926 21:32:14.790928 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:32:14.795141 (+ 4213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:32:14.796293 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:32:14.796299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:32:14.796327 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:32:14.796850 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.796870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.798540 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.798546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.801754 (+ 3208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:14.801772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.804034 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.804041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:14.804218 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.804250 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:32:14.804342 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 21:32:14.804503 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21084,"thread_start_us":20943,"threads_started":1}
I20260926 21:32:14.805315 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:14.770061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.97.1:41813)
0926 21:32:14.785203 (+ 15142us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:14.785217 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:14.791053 (+ 5836us) server_negotiation.cc:407] Connection header received
0926 21:32:14.795395 (+ 4342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:14.795405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:14.795673 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:14.795860 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:32:14.797039 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.798369 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.801987 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:14.803505 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:14.803656 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:14.803889 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:14.804772 (+ 883us) server_negotiation.cc:299] Negotiation successful
0926 21:32:14.804947 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14887,"thread_start_us":15925,"threads_started":1}
I20260926 21:32:14.805465 11983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22908c80194e42408d17675228e31811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:32:14.809538 12201 tablet_bootstrap.cc:492] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Bootstrap starting.
I20260926 21:32:14.809373 11840 tablet_service.cc:1467] Processing CreateTablet for tablet 22908c80194e42408d17675228e31811 (DEFAULT_TABLE table=test-table [id=0f4da8d2263841b6914895457b519c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:32:14.811017 11840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22908c80194e42408d17675228e31811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:32:14.814095 12201 tablet_bootstrap.cc:654] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Neither blocks nor log segments found. Creating new log.
I20260926 21:32:14.815397 12201 log.cc:824] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Log is configured to *not* fsync() on all Append() calls
I20260926 21:32:14.818797 12201 tablet_bootstrap.cc:492] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: No bootstrap required, opened a new log
I20260926 21:32:14.819097 12201 ts_tablet_manager.cc:1397] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260926 21:32:14.820684 12203 tablet_bootstrap.cc:492] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Bootstrap starting.
I20260926 21:32:14.825201 12203 tablet_bootstrap.cc:654] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Neither blocks nor log segments found. Creating new log.
I20260926 21:32:14.826414 12204 tablet_bootstrap.cc:492] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Bootstrap starting.
I20260926 21:32:14.826562 12203 log.cc:824] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Log is configured to *not* fsync() on all Append() calls
I20260926 21:32:14.831086 12204 tablet_bootstrap.cc:654] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Neither blocks nor log segments found. Creating new log.
I20260926 21:32:14.831285 12201 raft_consensus.cc:348] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:14.831781 12201 raft_consensus.cc:374] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:32:14.831882 12201 raft_consensus.cc:729] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3e29d3d6894d519c0dccb0c9e34e30, State: Initialized, Role: FOLLOWER
I20260926 21:32:14.831928 12203 tablet_bootstrap.cc:492] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: No bootstrap required, opened a new log
I20260926 21:32:14.832202 12203 ts_tablet_manager.cc:1397] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260926 21:32:14.832688 12201 consensus_queue.cc:260] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:14.834028 12204 log.cc:824] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Log is configured to *not* fsync() on all Append() calls
I20260926 21:32:14.843521 12182 heartbeater.cc:503] Master 127.0.97.62:39971 was elected leader, sending a full tablet report...
I20260926 21:32:14.843935 12201 ts_tablet_manager.cc:1428] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Time spent starting tablet: real 0.025s user 0.015s sys 0.007s
I20260926 21:32:14.845078 12204 tablet_bootstrap.cc:492] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: No bootstrap required, opened a new log
I20260926 21:32:14.845458 12204 ts_tablet_manager.cc:1397] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.006s
I20260926 21:32:14.845359 12203 raft_consensus.cc:348] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:14.845850 12203 raft_consensus.cc:374] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:32:14.845965 12203 raft_consensus.cc:729] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1c42c82cfe476086a5c81003e31b78, State: Initialized, Role: FOLLOWER
I20260926 21:32:14.846624 12203 consensus_queue.cc:260] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:14.847772 12048 heartbeater.cc:503] Master 127.0.97.62:39971 was elected leader, sending a full tablet report...
I20260926 21:32:14.850122 12203 ts_tablet_manager.cc:1428] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260926 21:32:14.855566 12204 raft_consensus.cc:348] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:14.855988 12204 raft_consensus.cc:374] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:32:14.856103 12204 raft_consensus.cc:729] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5627ab1f5d474fa599335f6ceaf9ae49, State: Initialized, Role: FOLLOWER
I20260926 21:32:14.856680 12204 consensus_queue.cc:260] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:14.858110 12204 ts_tablet_manager.cc:1428] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260926 21:32:14.876655 11906 tablet.cc:2369] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:32:14.942740 12183 tablet.cc:2369] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:32:15.013135 12206 raft_consensus.cc:482] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:32:15.013453 12206 raft_consensus.cc:504] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:15.015216 12206 leader_election.cc:288] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5627ab1f5d474fa599335f6ceaf9ae49 (127.0.97.1:41813), 0e1c42c82cfe476086a5c81003e31b78 (127.0.97.2:39883)
I20260926 21:32:15.022141 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:15.015451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39883 (local address 127.0.97.3:55337)
0926 21:32:15.015662 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 21:32:15.015682 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:32:15.015820 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:32:15.016239 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:32:15.016245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:32:15.016278 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:32:15.016486 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:15.016500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.017733 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.017736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:15.019830 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:15.019845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.021434 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.021438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:15.021619 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:15.021661 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 21:32:15.021758 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 21:32:15.021854 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":1}
I20260926 21:32:15.022281 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:15.015664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55337 (local address 127.0.97.2:39883)
0926 21:32:15.015878 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:15.015887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:15.015911 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:32:15.015989 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:15.015997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:15.016112 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:15.016211 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:32:15.016642 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.017605 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.020029 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.021268 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.021434 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:15.021550 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:15.021820 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 21:32:15.021968 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:32:15.022753 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:15.015893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34215 (local address 127.0.97.1:41813)
0926 21:32:15.016106 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:15.016114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:15.016539 (+ 425us) server_negotiation.cc:407] Connection header received
0926 21:32:15.016664 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:15.016671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:15.016754 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:15.016849 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:32:15.017349 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.018246 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.020500 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.021876 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.022006 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:15.022113 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:15.022488 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 21:32:15.022591 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:32:15.023304 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:15.015661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41813 (local address 127.0.97.3:34215)
0926 21:32:15.016416 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 21:32:15.016436 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:32:15.016547 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:32:15.016867 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:32:15.016870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:32:15.016888 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:32:15.017191 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:15.017200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.018373 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.018378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:15.020273 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:32:15.020283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.022090 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.022096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:32:15.022273 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:15.022307 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:32:15.022944 (+ 637us) client_negotiation.cc:240] Negotiation successful
0926 21:32:15.023090 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":423,"threads_started":1}
I20260926 21:32:15.023236 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22908c80194e42408d17675228e31811" candidate_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1c42c82cfe476086a5c81003e31b78" is_pre_election: true
I20260926 21:32:15.024171 12003 raft_consensus.cc:2454] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3e29d3d6894d519c0dccb0c9e34e30 in term 0.
I20260926 21:32:15.024938 12070 leader_election.cc:302] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1c42c82cfe476086a5c81003e31b78, fb3e29d3d6894d519c0dccb0c9e34e30; no voters:
I20260926 21:32:15.024868 11860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22908c80194e42408d17675228e31811" candidate_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" is_pre_election: true
I20260926 21:32:15.025331 12206 raft_consensus.cc:2781] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:32:15.025467 12206 raft_consensus.cc:482] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:32:15.025414 11860 raft_consensus.cc:2454] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3e29d3d6894d519c0dccb0c9e34e30 in term 0.
I20260926 21:32:15.025584 12206 raft_consensus.cc:3037] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:32:15.027882 12206 raft_consensus.cc:504] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:15.028556 12206 leader_election.cc:288] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [CANDIDATE]: Term 1 election: Requested vote from peers 5627ab1f5d474fa599335f6ceaf9ae49 (127.0.97.1:41813), 0e1c42c82cfe476086a5c81003e31b78 (127.0.97.2:39883)
I20260926 21:32:15.029125 11860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22908c80194e42408d17675228e31811" candidate_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5627ab1f5d474fa599335f6ceaf9ae49"
I20260926 21:32:15.029165 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22908c80194e42408d17675228e31811" candidate_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1c42c82cfe476086a5c81003e31b78"
I20260926 21:32:15.029307 11860 raft_consensus.cc:3037] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:32:15.029405 12003 raft_consensus.cc:3037] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:32:15.031632 11860 raft_consensus.cc:2454] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3e29d3d6894d519c0dccb0c9e34e30 in term 1.
I20260926 21:32:15.031872 12003 raft_consensus.cc:2454] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3e29d3d6894d519c0dccb0c9e34e30 in term 1.
I20260926 21:32:15.032096 12072 leader_election.cc:302] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5627ab1f5d474fa599335f6ceaf9ae49, fb3e29d3d6894d519c0dccb0c9e34e30; no voters:
I20260926 21:32:15.032401 12206 raft_consensus.cc:2781] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:32:15.032861 12206 raft_consensus.cc:686] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 1 LEADER]: Becoming Leader. State: Replica: fb3e29d3d6894d519c0dccb0c9e34e30, State: Running, Role: LEADER
I20260926 21:32:15.033368 12206 consensus_queue.cc:237] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } }
I20260926 21:32:15.038800 11691 catalog_manager.cc:5495] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb3e29d3d6894d519c0dccb0c9e34e30 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3e29d3d6894d519c0dccb0c9e34e30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 } health_report { overall_health: UNKNOWN } } }
I20260926 21:32:15.080443 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:15.074913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.0.97.3:36957)
0926 21:32:15.075086 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:15.075091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:15.075188 (+ 97us) server_negotiation.cc:407] Connection header received
0926 21:32:15.075369 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:15.075372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:15.075458 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:15.075615 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:32:15.076044 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.076734 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.078292 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.078754 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.078845 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:15.079322 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:32:15.079835 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:32:15.079941 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:15.080161 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 21:32:15.080224 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:32:15.088311 12003 raft_consensus.cc:1261] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 1 FOLLOWER]: Refusing update from remote peer fb3e29d3d6894d519c0dccb0c9e34e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:32:15.088364 11860 raft_consensus.cc:1261] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 1 FOLLOWER]: Refusing update from remote peer fb3e29d3d6894d519c0dccb0c9e34e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:32:15.089289 12211 consensus_queue.cc:1035] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5627ab1f5d474fa599335f6ceaf9ae49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:32:15.089659 12206 consensus_queue.cc:1035] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1c42c82cfe476086a5c81003e31b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:32:15.100611 12049 tablet.cc:2369] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:32:15.125742 11692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:32:15.126319 11692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:32:15.130771 11692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 745f5db686434b23840f08ecb8b61da1: Sending DeleteTablet for 3 replicas of tablet 22908c80194e42408d17675228e31811
I20260926 21:32:15.132102 11983 tablet_service.cc:1514] Processing DeleteTablet for tablet 22908c80194e42408d17675228e31811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:32:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37738
I20260926 21:32:15.132112 11840 tablet_service.cc:1514] Processing DeleteTablet for tablet 22908c80194e42408d17675228e31811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:32:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35452
I20260926 21:32:15.132838 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet 22908c80194e42408d17675228e31811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:32:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33110
I20260926 21:32:15.134349 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 11772
I20260926 21:32:15.142971 12225 tablet_replica.cc:330] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: stopping tablet replica
I20260926 21:32:15.143611 12225 raft_consensus.cc:2229] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:32:15.144562 12225 raft_consensus.cc:2258] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:32:15.145648 12223 tablet_replica.cc:330] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: stopping tablet replica
I20260926 21:32:15.146242 12223 raft_consensus.cc:2229] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:32:15.146987 12223 raft_consensus.cc:2258] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:32:15.148154 12224 tablet_replica.cc:330] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: stopping tablet replica
I20260926 21:32:15.148186 12225 ts_tablet_manager.cc:1905] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:32:15.149528 12223 ts_tablet_manager.cc:1905] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:32:15.150884 12224 raft_consensus.cc:2229] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:32:15.151528 12224 raft_consensus.cc:2258] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:32:15.153844 12224 ts_tablet_manager.cc:1905] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:32:15.158118 12223 ts_tablet_manager.cc:1918] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:32:15.158327 12223 log.cc:1196] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/22908c80194e42408d17675228e31811
I20260926 21:32:15.158957 12223 ts_tablet_manager.cc:1939] T 22908c80194e42408d17675228e31811 P 5627ab1f5d474fa599335f6ceaf9ae49: Deleting consensus metadata
I20260926 21:32:15.161218 11678 catalog_manager.cc:4841] TS 5627ab1f5d474fa599335f6ceaf9ae49 (127.0.97.1:41813): tablet 22908c80194e42408d17675228e31811 (table test-table [id=0f4da8d2263841b6914895457b519c3b]) successfully deleted
I20260926 21:32:15.162551 12224 ts_tablet_manager.cc:1918] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:32:15.162732 12224 log.cc:1196] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/22908c80194e42408d17675228e31811
I20260926 21:32:15.163275 12224 ts_tablet_manager.cc:1939] T 22908c80194e42408d17675228e31811 P 0e1c42c82cfe476086a5c81003e31b78: Deleting consensus metadata
I20260926 21:32:15.164036 12225 ts_tablet_manager.cc:1918] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:32:15.164243 12225 log.cc:1196] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/22908c80194e42408d17675228e31811
I20260926 21:32:15.164839 12225 ts_tablet_manager.cc:1939] T 22908c80194e42408d17675228e31811 P fb3e29d3d6894d519c0dccb0c9e34e30: Deleting consensus metadata
Sep 26 21:32:15 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:32:15.165325 11676 catalog_manager.cc:4841] TS 0e1c42c82cfe476086a5c81003e31b78 (127.0.97.2:39883): tablet 22908c80194e42408d17675228e31811 (table test-table [id=0f4da8d2263841b6914895457b519c3b]) successfully deleted
I20260926 21:32:15.166886 11678 catalog_manager.cc:4841] TS fb3e29d3d6894d519c0dccb0c9e34e30 (127.0.97.3:36957): tablet 22908c80194e42408d17675228e31811 (table test-table [id=0f4da8d2263841b6914895457b519c3b]) successfully deleted
I20260926 21:32:15.173166 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:15.135806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.0.97.1:41813)
0926 21:32:15.135987 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:15.135996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:15.142483 (+ 6487us) server_negotiation.cc:407] Connection header received
0926 21:32:15.143367 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:15.143373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:15.143484 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:15.143617 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:32:15.155506 (+ 11889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.156778 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.159701 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.160398 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.160519 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:15.166616 (+ 6097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:32:15.166637 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:32:15.166660 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:32:15.166715 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:32:15.169916 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:15.170806 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:15.170815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:15.170817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:15.170969 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:15.171398 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:15.171410 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:15.171413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:15.171803 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:32:15.172133 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:15.172556 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 21:32:15.172793 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:32:15.173872 11900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:32:15.269883 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 11772
I20260926 21:32:15.287590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 11911
Sep 26 21:32:15 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:32:15.305792 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:15.288532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.0.97.2:39883)
0926 21:32:15.288733 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:15.288738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:15.288757 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:32:15.288856 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:15.288860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:15.288938 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:15.289029 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:32:15.290678 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.291477 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.293303 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.293780 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.293894 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:15.299364 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:32:15.299381 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:32:15.299398 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:32:15.299442 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:32:15.302589 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:15.303520 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:15.303528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:15.303531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:15.303675 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:15.304071 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:15.304074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:15.304076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:15.304437 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:32:15.304667 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:15.305009 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 21:32:15.305212 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:32:15.306622 12043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:32:15.396898 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 11911
I20260926 21:32:15.413947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 12054
Sep 26 21:32:15 dist-test-slave-87bt krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:32:15.431447 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:32:15.415079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.0.97.3:36957)
0926 21:32:15.415397 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 21:32:15.415403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:32:15.415426 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:32:15.415482 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:32:15.415485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:32:15.415586 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:32:15.415702 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:32:15.417215 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.418123 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.419542 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:32:15.420031 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:32:15.420191 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:32:15.424969 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:32:15.424983 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:32:15.424999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:32:15.425045 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:32:15.428223 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:15.429074 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:15.429086 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:15.429088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:15.429241 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:32:15.429684 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:32:15.429690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:32:15.429693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:32:15.430133 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:32:15.430526 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0926 21:32:15.430864 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 21:32:15.431089 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260926 21:32:15.432128 12177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:32:15.555619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 12054
I20260926 21:32:15.575107 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 11660
I20260926 21:32:15.576416 11722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:32:15.687054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 11660
I20260926 21:32:15.707844 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 21:32:16.043924 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 21:32:16.044160 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 21:32:16.389870 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 21:32:16.390 UTC [11246] LOG: received smart shutdown request
2026-09-26 21:32:16.393 UTC [11246] LOG: background worker "logical replication launcher" (PID 11255) exited with exit code 1
2026-09-26 21:32:16.393 UTC [11250] LOG: shutting down
2026-09-26 21:32:16.573 UTC [11246] LOG: database system is shut down
2026-09-26T21:32:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50993 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:32:16 dist-test-slave-87bt krb5kdc[12239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:32:16 dist-test-slave-87bt krb5kdc[12239](info): set up 2 sockets
Sep 26 21:32:16 dist-test-slave-87bt krb5kdc[12239](info): commencing operation
krb5kdc: starting...
W20260926 21:32:17.810809 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:32:17 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:32:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 21:32:18.137099 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUh_ENg/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/postgres -l logfile start
2026-09-26 21:32:20.869 UTC [12269] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 21:32:20.869 UTC [12269] LOG: listening on IPv4 address "127.0.97.20", port 60315
2026-09-26 21:32:20.873 UTC [12269] LOG: listening on Unix socket "/tmp/.s.PGSQL.60315"
2026-09-26 21:32:20.896 UTC [12272] LOG: database system was shut down at 2026-09-26 21:32:19 UTC
2026-09-26 21:32:20.900 UTC [12269] LOG: database system is ready to accept connections
I20260926 21:32:21.886350 388 mini_postgres.cc:96] Postgres bound to 60315
2026-09-26 21:32:21.903 UTC [12280] FATAL: database "slave" does not exist
127.0.97.20:60315 - accepting connections
I20260926 21:32:21.903980 388 mini_ranger.cc:161] Starting Ranger...
I20260926 21:32:21.930511 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 21:32:22.432353 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 21:32:22.432484 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-admin
2026-09-26 21:32:22,726 [I] DB FLAVOR :POSTGRES
2026-09-26 21:32:22,726 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 21:32:22,727 [I] Checking connection..
2026-09-26 21:32:22,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 21:32:22,979 [I] Checking connection passed.
2026-09-26 21:32:22,979 [I] --------- Verifying version history table ---------
2026-09-26 21:32:22,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:32:23,246 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 21:32:23,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:32:23,504 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 21:32:23,504 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 21:32:23,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 21:32:23.743 UTC [12344] WARNING: there is no transaction in progress
2026-09-26 21:32:23,749 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 21:32:23,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:32:24,012 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 21:32:24,013 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 21:32:24,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 21:32:24,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 21:32:24,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-87bt.c.gcp-upstream.internal','N') ;"
2026-09-26 21:32:24,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 21:32:24,997 [I] Table x_portal_user does not exist in database ranger
2026-09-26 21:32:24,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 21:32:25,246 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 21:32:25,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 21:32:25,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 21:32:25,727 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 21:32:25,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 21:32:26.793 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.815 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.843 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.862 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.879 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.897 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.961 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.969 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.984 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:26.995 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:27.006 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:27.023 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:27.034 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:27.043 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:27.052 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:27.061 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 21:32:27,822 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 21:32:27,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 21:32:28,069 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 21:32:28,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 21:32:28,337 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 21:32:28,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 21:32:28,608 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 21:32:28,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 21:32:28,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 21:32:29,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-87bt.c.gcp-upstream.internal';"
2026-09-26 21:32:29,378 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 21:32:29,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 21:32:29,616 [I] DB_PATCHES have already been applied
I20260926 21:32:29.622658 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUh_ENg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 21:32:29.622773 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 21:32:29.637236 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:41895
Sep 26, 2026 9:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:32:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 21:32:30.784677 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.881s user 0.000s sys 0.039s
Sep 26, 2026 9:32:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:32:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 9:32:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:32:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:32:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:32:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 21:32:53.616917 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 21:32:53 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458373, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 21:32:53.805900 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 21:32:53.826871 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 21:32:54.742583 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 21:32:54.742703 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms
I20260926 21:32:54.742753 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:60315
2026-09-26 21:32:55,093 [I] DB FLAVOR :POSTGRES
2026-09-26 21:32:55,093 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 21:32:55,093 [I] Checking connection
2026-09-26 21:32:55,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 21:32:55,344 [I] connection success
2026-09-26 21:32:55,345 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 21:32:55,345 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 21:32:55,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 21:32:55,607 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 21:32:55,608 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 21:32:55,608 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 21:32:55,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:60315/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 21:32:55,893 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 21:32:56.383847 388 mini_ranger_kms.cc:310] Created kms service
I20260926 21:32:56.636313 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 21:32:56.687353 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 21:32:56 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458373, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:32:57.132118 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 21:32:57.132473 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUh_ENg/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUh_ENg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 21:32:57.132544 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 21:32:57.147708 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12633
I20260926 21:32:57.147868 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:51735
Sep 26, 2026 9:32:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:32:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:32:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:32:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:32:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:32:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 21:32:59.441293 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.635s user 0.087s sys 0.277s
I20260926 21:32:59.441555 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 21:32:59.441733 388 mini_ranger_kms.cc:401] 127.0.97.20:51735/kms/v1/keys
Sep 26 21:32:59 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 21:33:00 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:33:00 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:33:02 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:33:02.810317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46049
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46049
--ranger_config_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51735/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:03.153268 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:03.153571 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:03.153743 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:03.161226 12683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:03.161346 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:03.161446 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:03.161541 12683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:03.161608 12683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:03.172060 12683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:51735/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46049
--ranger_config_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:03.173442 12683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:03.174880 12683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:03.187460 12688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:03.187412 12691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:03.187491 12689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:03.188233 12683 server_base.cc:1034] running on GCE node
I20260926 21:33:03.189188 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:03.190791 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:03.192058 12683 hybrid_clock.cc:648] HybridClock initialized: now 1790458383192005 us; error 41 us; skew 500 ppm
I20260926 21:33:03.192492 12683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:03 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:03.203414 12683 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:33:03.205133 12683 webserver.cc:460] Webserver started at http://127.0.97.62:45277/ using document root <none> and password file <none>
I20260926 21:33:03.205906 12683 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:03.206058 12683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:03.206410 12683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:33:03 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:03.267005 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "f1461486aa2b42b8a773cb74ffddcbce"
format_stamp: "Formatted at 2026-09-26 21:33:03 on dist-test-slave-87bt"
server_key: "9064445995c2655a7c1fb494789f8632"
server_key_iv: "a75263d63f919060d2eb2b1217d07dd1"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:03.267861 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "f1461486aa2b42b8a773cb74ffddcbce"
format_stamp: "Formatted at 2026-09-26 21:33:03 on dist-test-slave-87bt"
server_key: "9064445995c2655a7c1fb494789f8632"
server_key_iv: "a75263d63f919060d2eb2b1217d07dd1"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:03.273541 12683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:33:03.299170 12698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:03.303699 12683 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.002s
I20260926 21:33:03.303968 12683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "f1461486aa2b42b8a773cb74ffddcbce"
format_stamp: "Formatted at 2026-09-26 21:33:03 on dist-test-slave-87bt"
server_key: "9064445995c2655a7c1fb494789f8632"
server_key_iv: "a75263d63f919060d2eb2b1217d07dd1"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:03.304316 12683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:03.325693 12683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:03.332070 12683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:03.332499 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:03.355345 12683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46049
I20260926 21:33:03.355445 12749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46049 every 8 connection(s)
I20260926 21:33:03.356885 12683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:33:03.358618 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 12683
I20260926 21:33:03.358830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:33:03.359647 388 external_mini_cluster.cc:1428] Setting key ba4e6e73bfe84f7056359ebe52b5ac18
I20260926 21:33:03.362169 12750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:33:03 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:03.375587 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Bootstrap starting.
I20260926 21:33:03.378964 12750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:03.379987 12750 log.cc:824] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:03.383482 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:03.361889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.0.97.62:46049)
0926 21:33:03.362717 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:03.362736 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:03.362777 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:33:03.364447 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:03.364469 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:03.364861 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:03.365186 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:03.366798 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:03.368192 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:03.370070 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:03.370816 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:03.370969 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:03.375557 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:03.375564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:03.375584 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:03.375620 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:03.378722 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:03.379504 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:03.379511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:03.379518 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:03.379677 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:03.380061 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:03.380066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:03.380070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:03.380878 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:03.381190 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:03.381704 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 21:33:03.382136 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":396,"threads_started":1}
I20260926 21:33:03.383566 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: No bootstrap required, opened a new log
I20260926 21:33:03.392496 12750 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } }
I20260926 21:33:03.392978 12750 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:03.393083 12750 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1461486aa2b42b8a773cb74ffddcbce, State: Initialized, Role: FOLLOWER
I20260926 21:33:03.393644 12750 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } }
I20260926 21:33:03.393874 12750 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:33:03.394028 12750 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:33:03.394212 12750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:03.396363 12750 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } }
I20260926 21:33:03.396878 12750 leader_election.cc:302] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1461486aa2b42b8a773cb74ffddcbce; no voters:
I20260926 21:33:03.397511 12750 leader_election.cc:288] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:33:03.397680 12755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:03.398386 12755 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 1 LEADER]: Becoming Leader. State: Replica: f1461486aa2b42b8a773cb74ffddcbce, State: Running, Role: LEADER
I20260926 21:33:03.398938 12755 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } }
I20260926 21:33:03.399822 12750 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:03.401925 12756 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1461486aa2b42b8a773cb74ffddcbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } } }
I20260926 21:33:03.402400 12756 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:03.403187 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1461486aa2b42b8a773cb74ffddcbce. Latest consensus state: current_term: 1 leader_uuid: "f1461486aa2b42b8a773cb74ffddcbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } } }
I20260926 21:33:03.403579 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:03.407527 12764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:33:03.413182 12764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:33:03.414052 12750 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 21:33:03.445335 12764 catalog_manager.cc:1347] Generated new cluster ID: 1536d9c3c44b4630881aa7b86ae71d08
I20260926 21:33:03.445498 12764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:33:03.457420 12764 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:33:03.459697 12764 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:33:03.477025 12764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Generated new TSK 0
I20260926 21:33:03.478077 12764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:33:04.295941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46049
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:33:04 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 21:33:04.691735 12795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:04.692097 12795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:04.692273 12795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:04.700207 12795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:04.700364 12795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:04.700605 12795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:33:04.712087 12795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:51735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:04.713500 12795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:04.715273 12795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:04.735386 12805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:04.735397 12806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:04.744670 12808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:04.746169 12795 server_base.cc:1034] running on GCE node
I20260926 21:33:04.747267 12795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:04.748550 12795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:04.749858 12795 hybrid_clock.cc:648] HybridClock initialized: now 1790458384749790 us; error 64 us; skew 500 ppm
I20260926 21:33:04.750280 12795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:04 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:04.755432 12795 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:33:04.760351 12795 webserver.cc:460] Webserver started at http://127.0.97.1:45287/ using document root <none> and password file <none>
I20260926 21:33:04.761384 12795 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:04.761541 12795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:04.761906 12795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:33:04 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:04.785076 12795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "0b5b3ca187074cb291fad7a7ae954bb7"
format_stamp: "Formatted at 2026-09-26 21:33:04 on dist-test-slave-87bt"
server_key: "a2efa6806aa6165ac5e8d23d84b1fa07"
server_key_iv: "42ff7ed24040de38f5f4bd085cfdba09"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:04.786221 12795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0b5b3ca187074cb291fad7a7ae954bb7"
format_stamp: "Formatted at 2026-09-26 21:33:04 on dist-test-slave-87bt"
server_key: "a2efa6806aa6165ac5e8d23d84b1fa07"
server_key_iv: "42ff7ed24040de38f5f4bd085cfdba09"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:04.802189 12795 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.011s sys 0.003s
I20260926 21:33:04.845930 12815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:04.847998 12795 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.004s
I20260926 21:33:04.848255 12795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "0b5b3ca187074cb291fad7a7ae954bb7"
format_stamp: "Formatted at 2026-09-26 21:33:04 on dist-test-slave-87bt"
server_key: "a2efa6806aa6165ac5e8d23d84b1fa07"
server_key_iv: "42ff7ed24040de38f5f4bd085cfdba09"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:04.848543 12795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:04.886127 12795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:04.891541 12795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:04.891901 12795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:04.894506 12795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:04.902729 12795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:04.902851 12795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:04.902992 12795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:04.903059 12795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:04.963722 12795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34559
I20260926 21:33:04.963742 12927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34559 every 8 connection(s)
I20260926 21:33:04.965628 12795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:33:04.970666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 12795
I20260926 21:33:04.970890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:33:04.971681 388 external_mini_cluster.cc:1428] Setting key 88c58caa408c3c70efc2f817ae9bd02d
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 26 21:33:05 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:05.045194 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:04.968639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60027 (local address 127.0.97.62:46049)
0926 21:33:04.989167 (+ 20528us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:04.989176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:05.003719 (+ 14543us) server_negotiation.cc:407] Connection header received
0926 21:33:05.021136 (+ 17417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:05.021144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:05.021263 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:05.021391 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:05.024943 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:05.026169 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:05.028912 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:05.029572 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:05.029693 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:05.036852 (+ 7159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:05.036861 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:05.036866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:05.036919 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:05.040585 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:05.041769 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:05.041776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:05.041779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:05.041972 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:05.042685 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:05.042691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:05.042695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:05.043113 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:05.043275 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:05.044446 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0926 21:33:05.044757 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20330,"thread_start_us":20532,"threads_started":1}
I20260926 21:33:05.048004 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:04.969446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46049 (local address 127.0.97.1:60027)
0926 21:33:05.003509 (+ 34063us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:05.003573 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:05.006256 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:05.022912 (+ 16656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:05.022929 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:05.023686 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:05.024631 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:05.024666 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:05.026476 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:05.026481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:05.028691 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:05.028709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:05.030058 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:05.030065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:05.030372 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:05.031396 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:05.031428 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:05.036487 (+ 5059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:05.040847 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:05.040857 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:05.040878 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:05.041587 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:05.042158 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:05.042163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:05.042174 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:05.042528 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:05.043379 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:05.043388 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:05.043935 (+ 547us) client_negotiation.cc:769] Sending connection context
0926 21:33:05.045760 (+ 1825us) client_negotiation.cc:240] Negotiation successful
0926 21:33:05.046445 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33659,"thread_start_us":36850,"threads_started":1}
I20260926 21:33:05.050359 12928 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46049
I20260926 21:33:05.050781 12928 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:05.051664 12928 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
I20260926 21:33:05.054303 12715 ts_manager.cc:194] Registered new tserver with Master: 0b5b3ca187074cb291fad7a7ae954bb7 (127.0.97.1:34559)
I20260926 21:33:05.056528 12715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260926 21:33:05.077834 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.102s user 0.000s sys 0.017s
I20260926 21:33:05.078840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46049
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:33:05 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:33:05 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 21:33:05.541571 12934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:05.542119 12934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:05.542320 12934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:05.554708 12934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:05.554981 12934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:05.555235 12934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:33:05.571244 12934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:51735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:05.573364 12934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:05.575295 12934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:05.613672 12942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:05.615398 12941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:05.635674 12944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:05.637574 12934 server_base.cc:1034] running on GCE node
I20260926 21:33:05.638837 12934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:05.647773 12934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:05.658574 12934 hybrid_clock.cc:648] HybridClock initialized: now 1790458385658442 us; error 117 us; skew 500 ppm
I20260926 21:33:05.659252 12934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:05 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:05.664126 12934 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:33:05.690192 12934 webserver.cc:460] Webserver started at http://127.0.97.2:34989/ using document root <none> and password file <none>
I20260926 21:33:05.691473 12934 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:05.691743 12934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:05.692176 12934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:33:05 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:05.719496 12934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "1af005fb4eba420da4390dc6fa296780"
format_stamp: "Formatted at 2026-09-26 21:33:05 on dist-test-slave-87bt"
server_key: "2055aa30840a167765aa2f2fb4e946a0"
server_key_iv: "95bb994ae217dcc0b6f66f598a876c22"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:05.720942 12934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1af005fb4eba420da4390dc6fa296780"
format_stamp: "Formatted at 2026-09-26 21:33:05 on dist-test-slave-87bt"
server_key: "2055aa30840a167765aa2f2fb4e946a0"
server_key_iv: "95bb994ae217dcc0b6f66f598a876c22"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:05.743806 12934 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.007s sys 0.011s
I20260926 21:33:05.793430 12954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:05.795908 12934 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.001s
I20260926 21:33:05.796244 12934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "1af005fb4eba420da4390dc6fa296780"
format_stamp: "Formatted at 2026-09-26 21:33:05 on dist-test-slave-87bt"
server_key: "2055aa30840a167765aa2f2fb4e946a0"
server_key_iv: "95bb994ae217dcc0b6f66f598a876c22"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:05.796993 12934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:05.833979 12934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:05.839749 12934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:05.840188 12934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:05.841513 12934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:05.843515 12934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:05.843626 12934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:05.843739 12934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:05.843798 12934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:05.877873 12934 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33675
I20260926 21:33:05.877892 13070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33675 every 8 connection(s)
I20260926 21:33:05.879624 12934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:33:05.889123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 12934
I20260926 21:33:05.889319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:33:05.889931 388 external_mini_cluster.cc:1428] Setting key 0a7f801aae203c5d4f8005059ec36c8a
Sep 26 21:33:05 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:05.903723 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:05.882458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40975 (local address 127.0.97.62:46049)
0926 21:33:05.882934 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:05.882939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:05.884088 (+ 1149us) server_negotiation.cc:407] Connection header received
0926 21:33:05.885998 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:05.886004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:05.886108 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:05.886225 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:05.887713 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:05.888452 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:05.890275 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:05.890874 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:05.891011 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:05.895292 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:05.895301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:05.895309 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:05.895372 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:05.899202 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:05.900218 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:05.900226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:05.900232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:05.900403 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:05.900998 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:05.901006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:05.901012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:05.901459 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:05.901614 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:05.902603 (+ 989us) server_negotiation.cc:299] Negotiation successful
0926 21:33:05.902993 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260926 21:33:05.904516 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:05.883148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46049 (local address 127.0.97.2:40975)
0926 21:33:05.883912 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:05.883956 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:05.885720 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:05.886522 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:05.886531 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:05.886966 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:05.887537 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:05.887554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:05.888602 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:05.888609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:05.890119 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:05.890138 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:05.891101 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:05.891109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:05.891354 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:05.892097 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:05.892140 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:05.895021 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:05.899388 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:05.899399 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:05.899418 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:05.900057 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:05.900532 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:05.900538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:05.900542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:05.900817 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:05.901601 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:05.901610 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:05.902064 (+ 454us) client_negotiation.cc:769] Sending connection context
0926 21:33:05.902375 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 21:33:05.902993 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":318,"threads_started":1}
I20260926 21:33:05.906183 13071 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46049
I20260926 21:33:05.906536 13071 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:05.907388 13071 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
I20260926 21:33:05.909129 12715 ts_manager.cc:194] Registered new tserver with Master: 1af005fb4eba420da4390dc6fa296780 (127.0.97.2:33675)
I20260926 21:33:05.910468 12715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40975
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:33:05.950716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46049
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:33:06.061364 12928 heartbeater.cc:503] Master 127.0.97.62:46049 was elected leader, sending a full tablet report...
W20260926 21:33:06.294133 13077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:06.294531 13077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:06.294693 13077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:06.302525 13077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:06.302685 13077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:06.302911 13077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:33:06.313572 13077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:51735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:06.315408 13077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:06.316716 13077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:06.331162 13082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:06.331111 13085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:06.331166 13083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:06.332258 13077 server_base.cc:1034] running on GCE node
I20260926 21:33:06.332978 13077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:06.334132 13077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:06.335407 13077 hybrid_clock.cc:648] HybridClock initialized: now 1790458386335320 us; error 82 us; skew 500 ppm
I20260926 21:33:06.335798 13077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:06 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:06.339375 13077 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:33:06.341019 13077 webserver.cc:460] Webserver started at http://127.0.97.3:34833/ using document root <none> and password file <none>
I20260926 21:33:06.341814 13077 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:06.341939 13077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:06.342234 13077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:33:06 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:06.360159 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "439f9552f2ed435fad22958424141f14"
format_stamp: "Formatted at 2026-09-26 21:33:06 on dist-test-slave-87bt"
server_key: "300fb134e85b3c2aeac8d49b97047e2e"
server_key_iv: "f7afd0e43d147654c08300c2ae13ea5a"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:06.361258 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "439f9552f2ed435fad22958424141f14"
format_stamp: "Formatted at 2026-09-26 21:33:06 on dist-test-slave-87bt"
server_key: "300fb134e85b3c2aeac8d49b97047e2e"
server_key_iv: "f7afd0e43d147654c08300c2ae13ea5a"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:06.368181 13077 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 21:33:06.383445 13092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:06.384984 13077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:33:06.385200 13077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "439f9552f2ed435fad22958424141f14"
format_stamp: "Formatted at 2026-09-26 21:33:06 on dist-test-slave-87bt"
server_key: "300fb134e85b3c2aeac8d49b97047e2e"
server_key_iv: "f7afd0e43d147654c08300c2ae13ea5a"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:06.385411 13077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:06.404030 13077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:06.408246 13077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:06.408514 13077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:06.409459 13077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:06.410986 13077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:06.411067 13077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:06.411170 13077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:06.411212 13077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:06.446871 13077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40601
I20260926 21:33:06.446902 13204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40601 every 8 connection(s)
I20260926 21:33:06.448334 13077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:33:06.456804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 13077
I20260926 21:33:06.457021 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:33:06.457652 388 external_mini_cluster.cc:1428] Setting key 1a259b1ec2711600c0e2feb1bd2e5404
Sep 26 21:33:06 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:06.471299 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:06.451245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47573 (local address 127.0.97.62:46049)
0926 21:33:06.451776 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:06.451781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:06.452872 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 21:33:06.454902 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:06.454908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:06.455030 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:06.455156 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:06.456937 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.458060 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.460408 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.461008 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.461155 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:06.464708 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:06.464714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:06.464717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:06.464756 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:06.467797 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:06.468756 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:06.468761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:06.468764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:06.468912 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:06.469392 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:06.469396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:06.469398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:06.469705 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:06.469833 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:06.470680 (+ 847us) server_negotiation.cc:299] Negotiation successful
0926 21:33:06.470902 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":284,"threads_started":1}
I20260926 21:33:06.472545 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:06.451708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46049 (local address 127.0.97.3:47573)
0926 21:33:06.452666 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:06.452716 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:06.454589 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:06.455426 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:06.455435 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:06.455899 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:06.456676 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:06.456698 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.458282 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.458288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:06.460252 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:06.460269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.461249 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.461254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:06.461467 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:06.462212 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:06.462238 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:06.464426 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:06.468004 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:06.468013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:06.468029 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:06.468553 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:06.469025 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:06.469028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:06.469030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:06.469243 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:06.469856 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:06.469863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:06.470229 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 21:33:06.470556 (+ 327us) client_negotiation.cc:240] Negotiation successful
0926 21:33:06.471008 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":515,"threads_started":1}
I20260926 21:33:06.474032 13205 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46049
I20260926 21:33:06.474391 13205 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:06.475216 13205 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
I20260926 21:33:06.476717 12714 ts_manager.cc:194] Registered new tserver with Master: 439f9552f2ed435fad22958424141f14 (127.0.97.3:40601)
I20260926 21:33:06.477725 12714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47573
I20260926 21:33:06.486670 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:33:06.488294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 12795
Sep 26 21:33:06 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:33:06.506153 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:06.489355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.0.97.1:34559)
0926 21:33:06.490018 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:06.490025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:06.490049 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:33:06.490110 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:06.490116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:06.490332 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:06.490539 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:06.492034 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.493050 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.494624 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.495200 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.495345 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:06.499810 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:06.499827 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:06.499844 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:06.499888 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:06.502978 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:06.503868 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:06.503876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:06.503878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:06.504035 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:06.504445 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:06.504449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:06.504451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:06.504879 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:06.505204 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:06.505521 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 21:33:06.505800 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":319,"threads_started":1}
I20260926 21:33:06.507537 12923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:06.592233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 12795
I20260926 21:33:06.609035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 12934
Sep 26 21:33:06 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:33:06.626210 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:06.610037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.97.2:33675)
0926 21:33:06.610774 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:06.610781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:06.610806 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:33:06.610878 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:06.610884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:06.611102 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:06.611235 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:06.612457 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.613475 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.615103 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.615736 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.615851 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:06.620241 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:06.620255 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:06.620271 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:06.620310 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:06.623189 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:06.623955 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:06.623963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:06.623966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:06.624129 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:06.624499 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:06.624503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:06.624505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:06.624915 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:06.625185 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:06.625494 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 21:33:06.625742 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":437,"threads_started":1}
I20260926 21:33:06.627650 13066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:06.710685 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 12934
I20260926 21:33:06.726341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 13077
Sep 26 21:33:06 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:33:06.742838 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:06.727345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.0.97.3:40601)
0926 21:33:06.727886 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:06.727892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:06.727921 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:33:06.727978 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:06.727984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:06.728227 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:06.728358 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:06.729551 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.730638 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.731924 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:06.732484 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:06.732581 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:06.736992 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:06.737006 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:06.737022 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:06.737062 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:06.740024 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:06.740715 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:06.740722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:06.740725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:06.740863 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:06.741205 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:06.741208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:06.741210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:06.741616 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:06.741859 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:06.742259 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 21:33:06.742549 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":272,"threads_started":1}
I20260926 21:33:06.744686 13200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:06.836829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 13077
I20260926 21:33:06.854322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 12683
I20260926 21:33:06.855681 12745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:06.966372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 12683
I20260926 21:33:06.990404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46049
--webserver_interface=127.0.97.62
--webserver_port=45277
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46049
--ranger_config_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51735/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51735/kms with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:07.326107 13217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:07.326469 13217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:07.326635 13217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:07.334193 13217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:07.334328 13217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:07.334403 13217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:07.334545 13217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:07.334635 13217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:07.345719 13217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51735/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46049
--ranger_config_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:07.347165 13217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:07.348578 13217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:07.361781 13223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:07.361816 13225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:07.361920 13222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:07.362716 13217 server_base.cc:1034] running on GCE node
I20260926 21:33:07.363591 13217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:07.365124 13217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:07.366369 13217 hybrid_clock.cc:648] HybridClock initialized: now 1790458387366318 us; error 52 us; skew 500 ppm
I20260926 21:33:07.366801 13217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:07 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:07.370041 13217 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:33:07.371775 13217 webserver.cc:460] Webserver started at http://127.0.97.62:45277/ using document root <none> and password file <none>
I20260926 21:33:07.372452 13217 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:07.372599 13217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 21:33:07 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:07.426319 13217 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 21:33:07.429651 13233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:07.431272 13217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:33:07.431495 13217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "f1461486aa2b42b8a773cb74ffddcbce"
format_stamp: "Formatted at 2026-09-26 21:33:03 on dist-test-slave-87bt"
server_key: "9064445995c2655a7c1fb494789f8632"
server_key_iv: "a75263d63f919060d2eb2b1217d07dd1"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:07.432560 13217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:07.466806 13217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:07.470772 13217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:07.471089 13217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:07.490352 13217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46049
I20260926 21:33:07.490331 13284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46049 every 8 connection(s)
I20260926 21:33:07.491991 13217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:33:07.494256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 13217
I20260926 21:33:07.495158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34559
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46049
--webserver_port=45287
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51735/kms with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:33:07.499236 13285 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:33:07.502753 13285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Bootstrap starting.
I20260926 21:33:07.522535 13285 log.cc:824] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:07.532066 13285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:33:07.532666 13285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Bootstrap complete.
I20260926 21:33:07.542477 13285 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } }
I20260926 21:33:07.543540 13285 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1461486aa2b42b8a773cb74ffddcbce, State: Initialized, Role: FOLLOWER
I20260926 21:33:07.544265 13285 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } }
I20260926 21:33:07.544543 13285 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:33:07.544687 13285 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:33:07.544876 13285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:33:07.549244 13285 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } }
I20260926 21:33:07.549798 13285 leader_election.cc:302] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1461486aa2b42b8a773cb74ffddcbce; no voters:
I20260926 21:33:07.550557 13285 leader_election.cc:288] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 21:33:07.550693 13289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:33:07.551923 13289 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [term 2 LEADER]: Becoming Leader. State: Replica: f1461486aa2b42b8a773cb74ffddcbce, State: Running, Role: LEADER
I20260926 21:33:07.552567 13289 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } }
I20260926 21:33:07.552935 13285 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:07.555119 13291 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1461486aa2b42b8a773cb74ffddcbce. Latest consensus state: current_term: 2 leader_uuid: "f1461486aa2b42b8a773cb74ffddcbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } } }
I20260926 21:33:07.555513 13291 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:07.556067 13290 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f1461486aa2b42b8a773cb74ffddcbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1461486aa2b42b8a773cb74ffddcbce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46049 } } }
I20260926 21:33:07.556329 13290 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:07.566452 13295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:33:07.573385 13295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:33:07.575269 13295 catalog_manager.cc:1259] Loaded cluster ID: 1536d9c3c44b4630881aa7b86ae71d08
I20260926 21:33:07.575487 13295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:33:07.580737 13295 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:33:07.582844 13295 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Loaded TSK: 0
I20260926 21:33:07.584997 13295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:33:07.591379 13285 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 21:33:07.965904 13287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:07.966413 13287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:07.966637 13287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:07.978998 13287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:07.979159 13287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:07.979422 13287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:33:07.996026 13287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45287
--tserver_master_addrs=127.0.97.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:07.997750 13287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:07.999568 13287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:08.051082 13322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:08.052763 13323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:08.062176 13287 server_base.cc:1034] running on GCE node
W20260926 21:33:08.062763 13326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:08.063854 13287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:08.067445 13287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:08.068840 13287 hybrid_clock.cc:648] HybridClock initialized: now 1790458388068758 us; error 76 us; skew 500 ppm
I20260926 21:33:08.069295 13287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:08 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:08.074043 13287 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:33:08.105808 13287 webserver.cc:460] Webserver started at http://127.0.97.1:45287/ using document root <none> and password file <none>
I20260926 21:33:08.107115 13287 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:08.107384 13287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 21:33:08 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:08.152428 13287 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:33:08.162974 13334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:08.164958 13287 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.003s
I20260926 21:33:08.165190 13287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "0b5b3ca187074cb291fad7a7ae954bb7"
format_stamp: "Formatted at 2026-09-26 21:33:04 on dist-test-slave-87bt"
server_key: "a2efa6806aa6165ac5e8d23d84b1fa07"
server_key_iv: "42ff7ed24040de38f5f4bd085cfdba09"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:08.166721 13287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:08.234319 13287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:08.239662 13287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:08.239998 13287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:08.241799 13287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:08.250025 13287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:08.250255 13287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 21:33:08.250458 13287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:08.250550 13287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:08.314774 13287 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34559
I20260926 21:33:08.314867 13446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34559 every 8 connection(s)
I20260926 21:33:08.316808 13287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:33:08.324534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 13287
I20260926 21:33:08.325232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33675
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46049
--webserver_port=34989
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51735/kms with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:33:08 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:08.362628 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:08.320482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46049 (local address 127.0.97.1:44667)
0926 21:33:08.331773 (+ 11291us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:08.331874 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:08.335285 (+ 3411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:08.338199 (+ 2914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:08.338211 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:08.338935 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:08.339805 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:08.339835 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:08.342584 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:08.342590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:08.344775 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:08.344805 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:08.346908 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:08.346916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:08.347238 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:08.348107 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:08.348153 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:08.351765 (+ 3612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:08.356851 (+ 5086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:08.356862 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:08.356882 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:08.357564 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:08.358187 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:08.358193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:08.358199 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:08.358508 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:08.359678 (+ 1170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:08.359687 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:08.360143 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 21:33:08.360487 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 21:33:08.361089 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10776,"thread_start_us":11995,"threads_started":1}
I20260926 21:33:08.364881 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:08.320378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44667 (local address 127.0.97.62:46049)
0926 21:33:08.334083 (+ 13705us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:08.334105 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:08.334174 (+ 69us) server_negotiation.cc:407] Connection header received
0926 21:33:08.336297 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:08.336320 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:08.336886 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:08.337376 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:08.340077 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:08.342348 (+ 2271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:08.345038 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:08.346221 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:08.346466 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:08.352169 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:08.352181 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:08.352205 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:08.352268 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:08.356596 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:08.357766 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:08.357777 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:08.357786 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:08.358019 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:08.358655 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:08.358660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:08.358664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:08.359377 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:08.359596 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:08.362916 (+ 3320us) server_negotiation.cc:299] Negotiation successful
0926 21:33:08.363421 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13405,"thread_start_us":13185,"threads_started":1}
I20260926 21:33:08.368471 13447 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46049
I20260926 21:33:08.368824 13447 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:08.369702 13447 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
W20260926 21:33:08.373392 13447 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46049 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 21:33:08.373632 13447 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:08.373971 13447 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
I20260926 21:33:08.375715 13447 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:08.376130 13447 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
W20260926 21:33:08.377476 13447 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:33:08.786935 13451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:08.787513 13451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:08.787768 13451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:08.800002 13451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:08.800206 13451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:08.800428 13451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:33:08.817713 13451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34989
--tserver_master_addrs=127.0.97.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:08.819518 13451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:08.821303 13451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:08.843987 13467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:08.844465 13468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 26 21:33:08 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 21:33:08.852756 13470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:08.854192 13451 server_base.cc:1034] running on GCE node
I20260926 21:33:08.855213 13451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:08.857893 13451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:08.859218 13451 hybrid_clock.cc:648] HybridClock initialized: now 1790458388859139 us; error 67 us; skew 500 ppm
I20260926 21:33:08.859654 13451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:08 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:08.864493 13451 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:33:08.884505 13451 webserver.cc:460] Webserver started at http://127.0.97.2:34989/ using document root <none> and password file <none>
I20260926 21:33:08.885687 13451 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:08.885862 13451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 21:33:08 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:08.921449 13451 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.004s
I20260926 21:33:08.929049 13478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:08.931041 13451 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260926 21:33:08.931281 13451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "1af005fb4eba420da4390dc6fa296780"
format_stamp: "Formatted at 2026-09-26 21:33:05 on dist-test-slave-87bt"
server_key: "2055aa30840a167765aa2f2fb4e946a0"
server_key_iv: "95bb994ae217dcc0b6f66f598a876c22"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:08.932397 13451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:08.953336 13451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:08.957962 13451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:08.958271 13451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:08.962939 13451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:08.965646 13451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:08.965781 13451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:08.965924 13451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:08.966010 13451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:09.014295 13451 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33675
I20260926 21:33:09.014393 13591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33675 every 8 connection(s)
I20260926 21:33:09.016224 13451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:33:09.017829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 13451
I20260926 21:33:09.018785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40601
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46049
--webserver_port=34833
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51735/kms with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:33:09 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:09.060788 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.026906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49123 (local address 127.0.97.62:46049)
0926 21:33:09.028017 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:09.028027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:09.030515 (+ 2488us) server_negotiation.cc:407] Connection header received
0926 21:33:09.032198 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:09.032203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:09.032307 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:09.032426 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:09.034608 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.035980 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.039188 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.039952 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.040077 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.051362 (+ 11285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:09.051373 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:09.051380 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:09.051439 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:09.056034 (+ 4595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:09.057215 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:09.057222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:09.057226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:09.057393 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:09.058159 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:09.058165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:09.058169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:09.058615 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:09.058756 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:09.060082 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0926 21:33:09.060382 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":815,"threads_started":1}
I20260926 21:33:09.063341 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.026207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46049 (local address 127.0.97.2:49123)
0926 21:33:09.029016 (+ 2809us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:09.029068 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:09.031890 (+ 2822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:09.032965 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:09.032980 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:09.033678 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:09.034338 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.034358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.036231 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.036239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.038998 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.039018 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.040497 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.040506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.040844 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.041695 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:09.041729 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:09.050933 (+ 9204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:09.056335 (+ 5402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:09.056346 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:09.056369 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:09.057046 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:09.057553 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:09.057558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:09.057563 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:09.058015 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:09.058858 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:09.058868 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:09.059525 (+ 657us) client_negotiation.cc:769] Sending connection context
0926 21:33:09.061077 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0926 21:33:09.061787 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2509,"thread_start_us":2287,"threads_started":1}
I20260926 21:33:09.065789 13592 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46049
I20260926 21:33:09.066134 13592 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:09.067113 13592 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
I20260926 21:33:09.069366 13250 ts_manager.cc:194] Registered new tserver with Master: 1af005fb4eba420da4390dc6fa296780 (127.0.97.2:33675)
I20260926 21:33:09.071831 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49123
Sep 26 21:33:09 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:09.377619 13447 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:09.378062 13447 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
I20260926 21:33:09.379444 13250 ts_manager.cc:194] Registered new tserver with Master: 0b5b3ca187074cb291fad7a7ae954bb7 (127.0.97.1:34559)
I20260926 21:33:09.380715 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44667
W20260926 21:33:09.488806 13595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:09.489204 13595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:09.489408 13595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:09.500540 13595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:09.500711 13595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:09.500896 13595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:33:09.517808 13595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34833
--tserver_master_addrs=127.0.97.62:46049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:09.519646 13595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:09.521595 13595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:09.540330 13602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:09.547051 13601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:09.549642 13595 server_base.cc:1034] running on GCE node
W20260926 21:33:09.549891 13604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:09.550864 13595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:09.552407 13595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:09.553723 13595 hybrid_clock.cc:648] HybridClock initialized: now 1790458389553651 us; error 65 us; skew 500 ppm
I20260926 21:33:09.554175 13595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:09 dist-test-slave-87bt krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:09.558943 13595 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:33:09.561296 13595 webserver.cc:460] Webserver started at http://127.0.97.3:34833/ using document root <none> and password file <none>
I20260926 21:33:09.562443 13595 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:09.562688 13595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 21:33:09 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:09.595538 13595 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.006s sys 0.005s
I20260926 21:33:09.599783 13612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:09.601841 13595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:33:09.602108 13595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "439f9552f2ed435fad22958424141f14"
format_stamp: "Formatted at 2026-09-26 21:33:06 on dist-test-slave-87bt"
server_key: "300fb134e85b3c2aeac8d49b97047e2e"
server_key_iv: "f7afd0e43d147654c08300c2ae13ea5a"
server_key_version: "kuduclusterkey@0"
I20260926 21:33:09.603649 13595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:09.632817 13595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:09.639128 13595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:09.639441 13595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:09.640686 13595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:09.642596 13595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:09.642711 13595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:09.642822 13595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:09.642899 13595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:33:09 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:33:09.805758 13595 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40601
I20260926 21:33:09.808048 13595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:33:09.808163 13725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40601 every 8 connection(s)
I20260926 21:33:09.809667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 13595
I20260926 21:33:09.846208 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.811604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.0.97.62:46049)
0926 21:33:09.820808 (+ 9204us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:09.820813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:09.820835 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:33:09.820906 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:09.820910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:09.820977 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:09.821141 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:09.823624 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.825160 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.829980 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.830946 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.831155 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.835073 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:09.835081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:09.835085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:09.835157 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:09.838902 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:09.843823 (+ 4921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:09.843832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:09.843836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:09.844027 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:09.844472 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:09.844477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:09.844482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:09.844892 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:09.845186 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:09.845549 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 21:33:09.845821 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9074,"thread_start_us":8960,"threads_started":1}
Sep 26 21:33:09 dist-test-slave-87bt krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:09.868144 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.823349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35257 (local address 127.0.97.62:46049)
0926 21:33:09.846848 (+ 23499us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:09.846854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:09.846876 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:33:09.846927 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:09.846931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:09.847024 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:09.847122 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:09.849596 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.850804 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.853486 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.854239 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.854367 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.859299 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:09.859307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:09.859312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:09.859359 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:09.863864 (+ 4505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:09.865064 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:09.865070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:09.865074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:09.865235 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:09.865849 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:09.865854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:09.865858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:09.866234 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:09.866390 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:09.867397 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0926 21:33:09.867672 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23348,"thread_start_us":26748,"threads_started":1}
I20260926 21:33:09.869416 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.822643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46049 (local address 127.0.97.3:35257)
0926 21:33:09.833150 (+ 10507us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:09.833211 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:09.841881 (+ 8670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:09.847672 (+ 5791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:09.847688 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:09.848380 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:09.849382 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.849415 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.851020 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.851026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.853289 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.853320 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.854622 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.854630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.854949 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.855983 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:09.856016 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:09.858938 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:09.864090 (+ 5152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:09.864100 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:09.864121 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:09.864888 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:09.865385 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:09.865390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:09.865394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:09.865693 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:09.866460 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:09.866470 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:09.866929 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 21:33:09.867278 (+ 349us) client_negotiation.cc:240] Negotiation successful
0926 21:33:09.867900 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10157,"thread_start_us":9899,"threads_started":1}
I20260926 21:33:09.871341 13726 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46049
I20260926 21:33:09.871711 13726 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:09.872668 13726 heartbeater.cc:511] Master 127.0.97.62:46049 requested a full tablet report, sending...
I20260926 21:33:09.875681 13250 ts_manager.cc:194] Registered new tserver with Master: 439f9552f2ed435fad22958424141f14 (127.0.97.3:40601)
I20260926 21:33:09.876801 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35257
I20260926 21:33:09.887907 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:33:09.915652 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.897490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.0.97.62:46049)
0926 21:33:09.897682 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:09.897688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:09.898187 (+ 499us) server_negotiation.cc:407] Connection header received
0926 21:33:09.898531 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:09.898538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:09.898623 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:09.898701 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:09.900833 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.902005 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.904246 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.904945 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.905099 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.908535 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:09.908542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:09.908547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:09.908594 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:09.912258 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:09.913253 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:09.913259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:09.913264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:09.913423 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:09.913898 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:09.913904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:09.913907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:09.914312 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:09.914618 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:09.914996 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 21:33:09.915280 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:33:09.922173 13250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:33:09.934113 13250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:33:09.976943 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.960218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:54338)
0926 21:33:09.967384 (+ 7166us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:09.967438 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:09.967669 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:09.968754 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:09.968762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:09.968796 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:09.969277 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.969293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.971485 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.971491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.973809 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.973832 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.975830 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.975837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.976060 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.976122 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 21:33:09.976252 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 21:33:09.976492 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6886,"thread_start_us":6749,"threads_started":1}
I20260926 21:33:09.978652 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.960398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.0.97.1:34559)
0926 21:33:09.967018 (+ 6620us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:09.967028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:09.967583 (+ 555us) server_negotiation.cc:407] Connection header received
0926 21:33:09.967947 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:09.967957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:09.968238 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:09.968448 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:09.969510 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.971314 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.973988 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.975558 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.977740 (+ 2182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.977993 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:09.978066 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 21:33:09.978239 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6365,"thread_start_us":6149,"threads_started":1}
I20260926 21:33:09.982563 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.968278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.0.97.3:40601)
0926 21:33:09.975179 (+ 6901us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:09.975189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:09.975222 (+ 33us) server_negotiation.cc:407] Connection header received
0926 21:33:09.975327 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:09.975336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:09.975615 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:09.975823 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:09.976771 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.978333 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.980331 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.981331 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.981497 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.981744 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:09.982018 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 21:33:09.982195 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6716,"thread_start_us":6571,"threads_started":1}
I20260926 21:33:09.983091 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.968195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40601 (local address 127.0.0.1:36188)
0926 21:33:09.974496 (+ 6301us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:09.974521 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:09.974642 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:09.976123 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:09.976127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:09.976155 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:09.976561 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.976572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.978496 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.978501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.980143 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.980156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.981580 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.981587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.981781 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.981830 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 21:33:09.982518 (+ 688us) client_negotiation.cc:240] Negotiation successful
0926 21:33:09.982714 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6115,"thread_start_us":5952,"threads_started":1}
I20260926 21:33:09.987524 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 97cae6fe2000479e88cae547b0cdb8d4 (DEFAULT_TABLE table=test-table [id=79d9ed8cd4fd438998ffdcc690dc4c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:09.987944 13660 tablet_service.cc:1467] Processing CreateTablet for tablet 97cae6fe2000479e88cae547b0cdb8d4 (DEFAULT_TABLE table=test-table [id=79d9ed8cd4fd438998ffdcc690dc4c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:09.989245 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cae6fe2000479e88cae547b0cdb8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:09.989300 13660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cae6fe2000479e88cae547b0cdb8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:09.989298 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.975280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33675 (local address 127.0.0.1:47006)
0926 21:33:09.977478 (+ 2198us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:09.977496 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:09.977586 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:09.983296 (+ 5710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:09.983300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:09.983324 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:09.983500 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.983510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.985283 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.985291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.986588 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:09.986601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.988526 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.988533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:09.988727 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.988764 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 21:33:09.988845 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 21:33:09.989030 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2079,"thread_start_us":7335,"threads_started":1}
I20260926 21:33:09.990327 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:09.974983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.0.97.2:33675)
0926 21:33:09.982171 (+ 7188us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:09.982179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:09.982207 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:33:09.982329 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:09.982337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:09.983016 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:09.983686 (+ 670us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:09.983858 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.985126 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.986742 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:09.988268 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:09.988399 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:09.989896 (+ 1497us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:09.989963 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 21:33:09.990085 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7025,"thread_start_us":6894,"threads_started":1}
I20260926 21:33:09.993690 13527 tablet_service.cc:1467] Processing CreateTablet for tablet 97cae6fe2000479e88cae547b0cdb8d4 (DEFAULT_TABLE table=test-table [id=79d9ed8cd4fd438998ffdcc690dc4c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:09.994911 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cae6fe2000479e88cae547b0cdb8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:10.005383 13754 tablet_bootstrap.cc:492] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Bootstrap starting.
I20260926 21:33:10.008015 13755 tablet_bootstrap.cc:492] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Bootstrap starting.
I20260926 21:33:10.009624 13756 tablet_bootstrap.cc:492] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: Bootstrap starting.
I20260926 21:33:10.011029 13754 tablet_bootstrap.cc:654] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:10.012235 13755 tablet_bootstrap.cc:654] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:10.013325 13754 log.cc:824] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:10.013713 13755 log.cc:824] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:10.013899 13756 tablet_bootstrap.cc:654] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:10.015580 13756 log.cc:824] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:10.024233 13754 tablet_bootstrap.cc:492] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: No bootstrap required, opened a new log
I20260926 21:33:10.024339 13755 tablet_bootstrap.cc:492] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: No bootstrap required, opened a new log
I20260926 21:33:10.024622 13754 ts_tablet_manager.cc:1397] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260926 21:33:10.024889 13755 ts_tablet_manager.cc:1397] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260926 21:33:10.034925 13754 raft_consensus.cc:348] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.035478 13754 raft_consensus.cc:374] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:10.035624 13754 raft_consensus.cc:729] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5b3ca187074cb291fad7a7ae954bb7, State: Initialized, Role: FOLLOWER
I20260926 21:33:10.036501 13754 consensus_queue.cc:260] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.037751 13756 tablet_bootstrap.cc:492] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: No bootstrap required, opened a new log
I20260926 21:33:10.037915 13447 heartbeater.cc:503] Master 127.0.97.62:46049 was elected leader, sending a full tablet report...
I20260926 21:33:10.038064 13756 ts_tablet_manager.cc:1397] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.011s
I20260926 21:33:10.038676 13754 ts_tablet_manager.cc:1428] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260926 21:33:10.038967 13755 raft_consensus.cc:348] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.039480 13755 raft_consensus.cc:374] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:10.039610 13755 raft_consensus.cc:729] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439f9552f2ed435fad22958424141f14, State: Initialized, Role: FOLLOWER
I20260926 21:33:10.040333 13755 consensus_queue.cc:260] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.041527 13726 heartbeater.cc:503] Master 127.0.97.62:46049 was elected leader, sending a full tablet report...
I20260926 21:33:10.042371 13755 ts_tablet_manager.cc:1428] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260926 21:33:10.048823 13756 raft_consensus.cc:348] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.049283 13756 raft_consensus.cc:374] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:10.049404 13756 raft_consensus.cc:729] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af005fb4eba420da4390dc6fa296780, State: Initialized, Role: FOLLOWER
I20260926 21:33:10.050092 13756 consensus_queue.cc:260] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.051347 13592 heartbeater.cc:503] Master 127.0.97.62:46049 was elected leader, sending a full tablet report...
I20260926 21:33:10.051831 13756 ts_tablet_manager.cc:1428] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260926 21:33:10.070892 13729 tablet.cc:2369] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:33:10.075642 13448 tablet.cc:2369] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:33:10.162062 13761 raft_consensus.cc:482] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:33:10.162372 13761 raft_consensus.cc:504] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.164145 13761 leader_election.cc:288] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b5b3ca187074cb291fad7a7ae954bb7 (127.0.97.1:34559), 1af005fb4eba420da4390dc6fa296780 (127.0.97.2:33675)
I20260926 21:33:10.171629 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:10.164675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59319 (local address 127.0.97.1:34559)
0926 21:33:10.164883 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:10.164890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:10.164910 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:33:10.165167 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:10.165173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:10.165274 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:10.165395 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:10.165908 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.166973 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.169404 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.170693 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.170835 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:10.170942 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:10.171220 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 21:33:10.171364 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:33:10.171650 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:10.164596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.97.3:59319)
0926 21:33:10.164808 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:10.164832 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:10.164999 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:10.165508 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:10.165513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:10.165539 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:10.165758 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:10.165773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.167111 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.167114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:10.169201 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:10.169219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.170865 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.170871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:10.171046 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:10.171073 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 21:33:10.171159 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 21:33:10.171319 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":49}
I20260926 21:33:10.172478 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:10.165192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38823 (local address 127.0.97.2:33675)
0926 21:33:10.165388 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:10.165396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:10.165813 (+ 417us) server_negotiation.cc:407] Connection header received
0926 21:33:10.165974 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:10.165979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:10.166068 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:10.166164 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:10.166715 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.167525 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.169875 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.171132 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.171280 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:10.171474 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:10.172106 (+ 632us) server_negotiation.cc:299] Negotiation successful
0926 21:33:10.172228 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:33:10.172969 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cae6fe2000479e88cae547b0cdb8d4" candidate_uuid: "439f9552f2ed435fad22958424141f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" is_pre_election: true
I20260926 21:33:10.173318 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:10.165033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33675 (local address 127.0.97.3:38823)
0926 21:33:10.165697 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:10.165718 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:10.165830 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:10.166193 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:10.166196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:10.166217 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:10.166564 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:10.166575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.167686 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.167691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:10.169626 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:10.169636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.171354 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.171359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:10.171546 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:10.171576 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 21:33:10.172724 (+ 1148us) client_negotiation.cc:240] Negotiation successful
0926 21:33:10.172864 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":2,"thread_start_us":362,"threads_started":1}
I20260926 21:33:10.173652 13402 raft_consensus.cc:2454] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 439f9552f2ed435fad22958424141f14 in term 0.
I20260926 21:33:10.174552 13616 leader_election.cc:302] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5b3ca187074cb291fad7a7ae954bb7, 439f9552f2ed435fad22958424141f14; no voters:
I20260926 21:33:10.174430 13547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cae6fe2000479e88cae547b0cdb8d4" candidate_uuid: "439f9552f2ed435fad22958424141f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af005fb4eba420da4390dc6fa296780" is_pre_election: true
I20260926 21:33:10.175050 13761 raft_consensus.cc:2781] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:33:10.175048 13547 raft_consensus.cc:2454] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 439f9552f2ed435fad22958424141f14 in term 0.
I20260926 21:33:10.175209 13761 raft_consensus.cc:482] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:33:10.175338 13761 raft_consensus.cc:3037] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:10.177625 13761 raft_consensus.cc:504] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.178228 13761 leader_election.cc:288] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [CANDIDATE]: Term 1 election: Requested vote from peers 0b5b3ca187074cb291fad7a7ae954bb7 (127.0.97.1:34559), 1af005fb4eba420da4390dc6fa296780 (127.0.97.2:33675)
I20260926 21:33:10.178778 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cae6fe2000479e88cae547b0cdb8d4" candidate_uuid: "439f9552f2ed435fad22958424141f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5b3ca187074cb291fad7a7ae954bb7"
I20260926 21:33:10.178853 13547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cae6fe2000479e88cae547b0cdb8d4" candidate_uuid: "439f9552f2ed435fad22958424141f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af005fb4eba420da4390dc6fa296780"
I20260926 21:33:10.178993 13402 raft_consensus.cc:3037] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:10.179129 13547 raft_consensus.cc:3037] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:10.181231 13402 raft_consensus.cc:2454] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439f9552f2ed435fad22958424141f14 in term 1.
I20260926 21:33:10.181277 13547 raft_consensus.cc:2454] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439f9552f2ed435fad22958424141f14 in term 1.
I20260926 21:33:10.181710 13616 leader_election.cc:302] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af005fb4eba420da4390dc6fa296780, 439f9552f2ed435fad22958424141f14; no voters:
I20260926 21:33:10.182066 13761 raft_consensus.cc:2781] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:10.182613 13761 raft_consensus.cc:686] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 1 LEADER]: Becoming Leader. State: Replica: 439f9552f2ed435fad22958424141f14, State: Running, Role: LEADER
I20260926 21:33:10.183141 13761 consensus_queue.cc:237] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } }
I20260926 21:33:10.188046 13250 catalog_manager.cc:5495] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 439f9552f2ed435fad22958424141f14 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "439f9552f2ed435fad22958424141f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "439f9552f2ed435fad22958424141f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 } health_report { overall_health: UNKNOWN } } }
I20260926 21:33:10.233256 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:10.227393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.0.97.3:40601)
0926 21:33:10.227603 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:10.227608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:10.227719 (+ 111us) server_negotiation.cc:407] Connection header received
0926 21:33:10.227861 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:10.227865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:10.227958 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:10.228105 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:33:10.228561 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.229311 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.230984 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.231413 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.231557 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:10.232095 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:33:10.232633 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:33:10.232807 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:10.232864 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 21:33:10.232964 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:33:10.242897 13547 raft_consensus.cc:1261] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 1 FOLLOWER]: Refusing update from remote peer 439f9552f2ed435fad22958424141f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:10.243028 13402 raft_consensus.cc:1261] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 1 FOLLOWER]: Refusing update from remote peer 439f9552f2ed435fad22958424141f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:10.244071 13761 consensus_queue.cc:1035] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af005fb4eba420da4390dc6fa296780" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:10.244508 13764 consensus_queue.cc:1035] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5b3ca187074cb291fad7a7ae954bb7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:10.258711 13767 mvcc.cc:205] Tried to move back new op lower bound from 7333717566422663168 to 7333717566194307072. Current Snapshot: MvccSnapshot[applied={T|T < 7333717566422663168}]
I20260926 21:33:10.279532 13250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:33:10.279835 13250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:33:10.284374 13250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1461486aa2b42b8a773cb74ffddcbce: Sending DeleteTablet for 3 replicas of tablet 97cae6fe2000479e88cae547b0cdb8d4
I20260926 21:33:10.285880 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cae6fe2000479e88cae547b0cdb8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54338
I20260926 21:33:10.286054 13527 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cae6fe2000479e88cae547b0cdb8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47006
I20260926 21:33:10.286141 13660 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cae6fe2000479e88cae547b0cdb8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36188
I20260926 21:33:10.291666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 13287
I20260926 21:33:10.297147 13777 tablet_replica.cc:330] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: stopping tablet replica
I20260926 21:33:10.297690 13777 raft_consensus.cc:2229] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:10.298309 13777 raft_consensus.cc:2258] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:10.300643 13779 tablet_replica.cc:330] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: stopping tablet replica
I20260926 21:33:10.301128 13779 raft_consensus.cc:2229] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:10.301635 13779 raft_consensus.cc:2258] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:10.304313 13777 ts_tablet_manager.cc:1905] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:10.311318 13779 ts_tablet_manager.cc:1905] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:10.311992 13777 ts_tablet_manager.cc:1918] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:10.312175 13777 log.cc:1196] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/97cae6fe2000479e88cae547b0cdb8d4
I20260926 21:33:10.312729 13777 ts_tablet_manager.cc:1939] T 97cae6fe2000479e88cae547b0cdb8d4 P 0b5b3ca187074cb291fad7a7ae954bb7: Deleting consensus metadata
I20260926 21:33:10.327685 13237 catalog_manager.cc:4841] TS 0b5b3ca187074cb291fad7a7ae954bb7 (127.0.97.1:34559): tablet 97cae6fe2000479e88cae547b0cdb8d4 (table test-table [id=79d9ed8cd4fd438998ffdcc690dc4c2c]) successfully deleted
I20260926 21:33:10.332214 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:10.293322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.0.97.1:34559)
0926 21:33:10.293588 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:10.293597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:10.294229 (+ 632us) server_negotiation.cc:407] Connection header received
0926 21:33:10.294540 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:10.294549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:10.294664 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:10.294795 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:10.297725 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.299035 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.308627 (+ 9592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.309314 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.309480 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:10.323870 (+ 14390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:10.323889 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:10.323909 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:10.323959 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:10.328209 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:10.329507 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:10.329516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:10.329520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:10.329696 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:10.330212 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:10.330217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:10.330220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:10.330748 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:10.331069 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:10.331549 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 21:33:10.331785 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":94}
I20260926 21:33:10.332858 13442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:10.334980 13779 ts_tablet_manager.cc:1918] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:10.335382 13779 log.cc:1196] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/97cae6fe2000479e88cae547b0cdb8d4
I20260926 21:33:10.336294 13779 ts_tablet_manager.cc:1939] T 97cae6fe2000479e88cae547b0cdb8d4 P 1af005fb4eba420da4390dc6fa296780: Deleting consensus metadata
I20260926 21:33:10.337085 13778 tablet_replica.cc:330] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: stopping tablet replica
I20260926 21:33:10.339534 13237 catalog_manager.cc:4841] TS 1af005fb4eba420da4390dc6fa296780 (127.0.97.2:33675): tablet 97cae6fe2000479e88cae547b0cdb8d4 (table test-table [id=79d9ed8cd4fd438998ffdcc690dc4c2c]) successfully deleted
I20260926 21:33:10.339586 13778 raft_consensus.cc:2229] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:33:10.340466 13778 raft_consensus.cc:2258] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:10.343215 13778 ts_tablet_manager.cc:1905] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:10.351186 13778 ts_tablet_manager.cc:1918] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:10.351394 13778 log.cc:1196] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/97cae6fe2000479e88cae547b0cdb8d4
I20260926 21:33:10.351977 13778 ts_tablet_manager.cc:1939] T 97cae6fe2000479e88cae547b0cdb8d4 P 439f9552f2ed435fad22958424141f14: Deleting consensus metadata
I20260926 21:33:10.353967 13237 catalog_manager.cc:4841] TS 439f9552f2ed435fad22958424141f14 (127.0.97.3:40601): tablet 97cae6fe2000479e88cae547b0cdb8d4 (table test-table [id=79d9ed8cd4fd438998ffdcc690dc4c2c]) successfully deleted
I20260926 21:33:10.436621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 13287
I20260926 21:33:10.459936 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 13451
I20260926 21:33:10.476536 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:10.460844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.0.97.2:33675)
0926 21:33:10.461061 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:10.461067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:10.461092 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:33:10.461162 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:10.461168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:10.461256 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:10.461372 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:10.462801 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.464005 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.465524 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.466182 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.466321 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:10.469170 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:10.469191 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:10.469212 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:10.469268 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:10.473224 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:10.474114 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:10.474123 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:10.474126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:10.474295 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:10.474783 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:10.474789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:10.474792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:10.475293 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:10.475592 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:10.475869 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 21:33:10.476148 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:33:10.477425 13587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:10.570698 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 13451
I20260926 21:33:10.588608 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 13595
I20260926 21:33:10.605372 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:10.589461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.0.97.3:40601)
0926 21:33:10.589654 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:10.589661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:10.589683 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:33:10.589833 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:10.589841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:10.589935 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:10.590086 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:10.591813 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.593015 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.594615 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:10.595315 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:10.595460 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:10.598021 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:10.598036 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:10.598056 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:10.598104 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:10.602183 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:10.602989 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:10.602999 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:10.603003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:10.603201 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:10.603661 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:10.603671 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:10.603675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:10.604176 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:10.604516 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:10.604727 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 21:33:10.604975 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:33:10.606242 13721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:10.725407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 13595
I20260926 21:33:10.744566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 13217
I20260926 21:33:10.745765 13280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:10.861599 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 13217
I20260926 21:33:10.887316 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 21:33:11.223220 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 21:33:11.223464 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 21:33:11.569492 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 21:33:11.569 UTC [12269] LOG: received smart shutdown request
2026-09-26 21:33:11.572 UTC [12269] LOG: background worker "logical replication launcher" (PID 12278) exited with exit code 1
2026-09-26 21:33:11.573 UTC [12273] LOG: shutting down
2026-09-26 21:33:11.763 UTC [12269] LOG: database system is shut down
2026-09-26T21:33:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55194 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:33:11 dist-test-slave-87bt krb5kdc[13792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:33:11 dist-test-slave-87bt krb5kdc[13792](info): set up 2 sockets
Sep 26 21:33:11 dist-test-slave-87bt krb5kdc[13792](info): commencing operation
krb5kdc: starting...
W20260926 21:33:13.002818 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:33:13 dist-test-slave-87bt krb5kdc[13792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:33:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:33:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:33:13.237635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44507
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44507,127.0.97.61:40499,127.0.97.60:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:13.594868 13805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:13.595207 13805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:13.595328 13805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:13.602923 13805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:13.603058 13805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:13.603111 13805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:13.603176 13805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:13.603228 13805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:13.613689 13805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44507,127.0.97.61:40499,127.0.97.60:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:13.615036 13805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:13.616329 13805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:13.628401 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:13.628604 13810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:13.628401 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:13.630227 13805 server_base.cc:1034] running on GCE node
I20260926 21:33:13.631150 13805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:13.632660 13805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:13.633898 13805 hybrid_clock.cc:648] HybridClock initialized: now 1790458393633861 us; error 39 us; skew 500 ppm
I20260926 21:33:13.634261 13805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:13 dist-test-slave-87bt krb5kdc[13792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:13.637568 13805 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:33:13.639191 13805 webserver.cc:460] Webserver started at http://127.0.97.62:46811/ using document root <none> and password file <none>
I20260926 21:33:13.639923 13805 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:13.640082 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:13.640398 13805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:13.643605 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "4a375cdcd8484fa7b49eebab781f0a08"
format_stamp: "Formatted at 2026-09-26 21:33:13 on dist-test-slave-87bt"
server_key: "8138bb9fddbb8382bbea249d349c14aa"
server_key_iv: "5e2e5e51375b915d67f57164c147ce05"
server_key_version: "encryptionkey@0"
I20260926 21:33:13.644449 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "4a375cdcd8484fa7b49eebab781f0a08"
format_stamp: "Formatted at 2026-09-26 21:33:13 on dist-test-slave-87bt"
server_key: "8138bb9fddbb8382bbea249d349c14aa"
server_key_iv: "5e2e5e51375b915d67f57164c147ce05"
server_key_version: "encryptionkey@0"
I20260926 21:33:13.650125 13805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:33:13.653951 13820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:13.655511 13805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:33:13.655750 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "4a375cdcd8484fa7b49eebab781f0a08"
format_stamp: "Formatted at 2026-09-26 21:33:13 on dist-test-slave-87bt"
server_key: "8138bb9fddbb8382bbea249d349c14aa"
server_key_iv: "5e2e5e51375b915d67f57164c147ce05"
server_key_version: "encryptionkey@0"
I20260926 21:33:13.655982 13805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:13.691738 13805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:13.696293 13805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:13.696612 13805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:13.715896 13805 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44507
I20260926 21:33:13.715927 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44507 every 8 connection(s)
I20260926 21:33:13.717326 13805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:33:13.720559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 13805
I20260926 21:33:13.720777 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:33:13.721308 13872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:13.721487 388 external_mini_cluster.cc:1428] Setting key ab1291b5f791a9a891c00eb71eb63e80
I20260926 21:33:13.728400 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 26 21:33:13 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:13.756903 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.732846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.0.97.62:44507)
0926 21:33:13.733952 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:13.733972 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:13.734533 (+ 561us) server_negotiation.cc:407] Connection header received
0926 21:33:13.737770 (+ 3237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:13.737776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:13.738023 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:13.738189 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:13.740016 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:13.741207 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:13.743573 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:13.744175 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:13.744330 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:13.748253 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:13.748266 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:13.748288 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:13.748338 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:13.752591 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:13.753351 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:13.753363 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:13.753370 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:13.753594 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:13.754087 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:13.754090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:13.754092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:13.754694 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:13.754854 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:13.755548 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 21:33:13.755815 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":509,"threads_started":1}
I20260926 21:33:13.756939 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.732845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44507 (local address 127.0.0.1:58774)
0926 21:33:13.734320 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.734368 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:13.737322 (+ 2954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:13.738298 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:13.738311 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:13.739241 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:13.739762 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:13.739795 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:13.741386 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:13.741393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:13.743406 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:13.743429 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:13.744493 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:13.744501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:13.744631 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:13.745360 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:13.745383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:13.747954 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:13.752756 (+ 4802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:13.752767 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:13.752778 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:13.753194 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:13.753740 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:13.753748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:13.753753 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:13.753976 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:13.754835 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:13.754846 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:13.755141 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 21:33:13.755421 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 21:33:13.756045 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"mutex_wait_us":50,"spinlock_wait_cycles":24448,"thread_start_us":882,"threads_started":1}
I20260926 21:33:13.760208 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:33:13.761394 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.760965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53548)
0926 21:33:13.761188 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.761288 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260926 21:33:13.761694 13823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
I20260926 21:33:13.763356 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.762985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53564)
0926 21:33:13.763135 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.763213 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:33:13.763540 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40499: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 21:33:13.777643 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40499
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44507,127.0.97.61:40499,127.0.97.60:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:33:13.812866 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } attempt: 1
I20260926 21:33:13.813985 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.813543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53576)
0926 21:33:13.813743 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.813853 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 21:33:13.815032 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.814650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53580)
0926 21:33:13.814826 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.814900 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 21:33:13.815227 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40499: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
I20260926 21:33:13.883474 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } attempt: 2
I20260926 21:33:13.884634 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.884142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53590)
0926 21:33:13.884408 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.884500 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 21:33:13.885625 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.885280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53604)
0926 21:33:13.885442 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.885503 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 21:33:13.885815 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40499: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
I20260926 21:33:13.977063 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } attempt: 3
I20260926 21:33:13.978163 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.977718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53620)
0926 21:33:13.977928 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.978023 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 21:33:13.979207 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:13.978845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53632)
0926 21:33:13.978989 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:13.979064 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 21:33:13.979413 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40499: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
W20260926 21:33:14.121073 13879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:14.121459 13879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:14.121628 13879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 21:33:14.122642 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } attempt: 4
I20260926 21:33:14.123873 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.123298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53636)
0926 21:33:14.123651 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.123747 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260926 21:33:14.124975 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.124618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53644)
0926 21:33:14.124765 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.124855 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:33:14.125175 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40499: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40499: connect: Connection refused (error 111)
W20260926 21:33:14.129524 13879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:14.129703 13879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:14.129808 13879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:14.129916 13879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:14.129992 13879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:14.140254 13879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44507,127.0.97.61:40499,127.0.97.60:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:14.141693 13879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:14.143082 13879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:14.156145 13887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:14.156176 13885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:14.156224 13884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:14.157073 13879 server_base.cc:1034] running on GCE node
I20260926 21:33:14.157964 13879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:14.159811 13879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:14.161067 13879 hybrid_clock.cc:648] HybridClock initialized: now 1790458394161008 us; error 62 us; skew 500 ppm
I20260926 21:33:14.161441 13879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:14.164670 13879 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 21:33:14.166188 13879 webserver.cc:460] Webserver started at http://127.0.97.61:36277/ using document root <none> and password file <none>
I20260926 21:33:14.166952 13879 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:14.167091 13879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:14.167407 13879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:14.170786 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/data/instance:
uuid: "c083a24c24c94cdd987ab961b485d582"
format_stamp: "Formatted at 2026-09-26 21:33:14 on dist-test-slave-87bt"
server_key: "dca85162de354e932200a5f1ea69b6d9"
server_key_iv: "b69877d69681542af655e0d50a2148dd"
server_key_version: "encryptionkey@0"
I20260926 21:33:14.171723 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/wal/instance:
uuid: "c083a24c24c94cdd987ab961b485d582"
format_stamp: "Formatted at 2026-09-26 21:33:14 on dist-test-slave-87bt"
server_key: "dca85162de354e932200a5f1ea69b6d9"
server_key_iv: "b69877d69681542af655e0d50a2148dd"
server_key_version: "encryptionkey@0"
I20260926 21:33:14.177132 13879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:33:14.180747 13894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:14.182161 13879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:33:14.182365 13879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/wal
uuid: "c083a24c24c94cdd987ab961b485d582"
format_stamp: "Formatted at 2026-09-26 21:33:14 on dist-test-slave-87bt"
server_key: "dca85162de354e932200a5f1ea69b6d9"
server_key_iv: "b69877d69681542af655e0d50a2148dd"
server_key_version: "encryptionkey@0"
I20260926 21:33:14.182689 13879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:14.204761 13879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:14.209717 13879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:14.210017 13879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:14.229872 13879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40499
I20260926 21:33:14.229982 13945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40499 every 8 connection(s)
I20260926 21:33:14.231328 13879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:33:14.235137 13946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:14.240432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 13879
I20260926 21:33:14.240628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-1/wal/instance
I20260926 21:33:14.241129 388 external_mini_cluster.cc:1428] Setting key f6827b48f41f64b9082a8fdbc0439cf3
I20260926 21:33:14.242709 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 21:33:14.267236 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.246482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.0.97.62:44507)
0926 21:33:14.246724 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.246731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.248374 (+ 1643us) server_negotiation.cc:407] Connection header received
0926 21:33:14.250485 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.250495 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.250603 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.250730 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.252375 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.253535 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.255252 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.255995 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.256161 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.259888 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.259898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.259906 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.259959 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.263853 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.264727 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.264731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.264733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.264858 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.265247 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.265250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.265252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.265624 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.265757 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.266745 (+ 988us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.266992 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 21:33:14.268647 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.247133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44507 (local address 127.0.0.1:58776)
0926 21:33:14.248139 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.248192 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:14.250090 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:14.250965 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:14.250974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:14.251566 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:14.252160 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.252177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.253721 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.253725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.255088 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.255100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.256234 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.256239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.256455 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.257205 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:14.257235 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:14.259632 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:14.264046 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.264054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.264069 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.264558 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.264958 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.264961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.264963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.265147 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.265754 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:14.265761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:14.266145 (+ 384us) client_negotiation.cc:769] Sending connection context
0926 21:33:14.266483 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 21:33:14.267044 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":373,"threads_started":1}
I20260926 21:33:14.270615 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:33:14.288772 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.271486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53646)
0926 21:33:14.271708 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.271730 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:14.271848 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:14.272852 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:14.272859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:14.273402 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:14.273621 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.273631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.275837 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.275841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.276678 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.276689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.278050 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.278054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.278193 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.278866 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:14.278884 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:14.281156 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:14.285959 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.285964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.285967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.286360 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.286964 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.286971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.286976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.287123 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.287775 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:14.287780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:14.287948 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 21:33:14.288084 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 21:33:14.288384 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 21:33:14.288900 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.271722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.0.97.61:40499)
0926 21:33:14.272393 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.272399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.272426 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:33:14.272470 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.272476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.272661 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.272808 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.273803 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.275654 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.276830 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.277877 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.278037 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.281424 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.281445 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.281469 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.281523 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.285807 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.286683 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.286698 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.286700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.286836 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.287229 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.287235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.287241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.287646 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.287881 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.288234 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.288516 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":506,"thread_start_us":371,"threads_started":1}
I20260926 21:33:14.291700 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:33:14.292894 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.292409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50406)
0926 21:33:14.292576 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.292746 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 21:33:14.293157 13898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
I20260926 21:33:14.294876 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.294530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50420)
0926 21:33:14.294686 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.294765 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 21:33:14.295117 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 21:33:14.297974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43825
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44507,127.0.97.61:40499,127.0.97.60:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:33:14.344516 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 1
I20260926 21:33:14.345726 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.345224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50424)
0926 21:33:14.345422 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.345536 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 21:33:14.346872 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.346508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50430)
0926 21:33:14.346685 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.346755 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 21:33:14.347100 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
I20260926 21:33:14.415366 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 2
I20260926 21:33:14.416693 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.416139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50434)
0926 21:33:14.416352 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.416464 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 21:33:14.417814 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.417426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50448)
0926 21:33:14.417601 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.417679 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 21:33:14.418058 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
I20260926 21:33:14.424461 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } attempt: 5
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:33:14.440001 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.425199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53656)
0926 21:33:14.425393 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.425413 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:14.425530 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:14.425984 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:14.425989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:14.426381 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:14.426691 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.426703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.428105 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.428110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.428963 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.428975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.430057 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.430062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.430223 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.431050 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:14.431079 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:14.434135 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:14.437421 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.437426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.437431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.437833 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.438251 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.438257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.438260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.438410 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.438996 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:14.439001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:14.439186 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 21:33:14.439332 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 21:33:14.439659 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 21:33:14.440001 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.425286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.0.97.61:40499)
0926 21:33:14.425472 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.425478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.425501 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:33:14.425691 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.425696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.425794 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.425915 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.426864 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.427936 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.429121 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.429859 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.429988 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.434478 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.434487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.434492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.434538 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.437237 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.437966 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.437972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.437976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.438130 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.438570 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.438576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.438579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.438872 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.439036 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.439456 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.439673 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:33:14.441555 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:33:14.442767 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.442313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50458)
0926 21:33:14.442525 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.442616 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 21:33:14.443904 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.443568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50474)
0926 21:33:14.443742 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.443810 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 21:33:14.444219 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
I20260926 21:33:14.495504 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 1
I20260926 21:33:14.496903 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.496363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50484)
0926 21:33:14.496612 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.496717 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 21:33:14.497995 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.497637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50494)
0926 21:33:14.497795 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.497879 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 21:33:14.498232 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
I20260926 21:33:14.509280 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 3
I20260926 21:33:14.510581 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.509995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50504)
0926 21:33:14.510251 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.510346 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 21:33:14.511723 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.511252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50512)
0926 21:33:14.511498 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.511585 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260926 21:33:14.511915 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
I20260926 21:33:14.566484 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 2
I20260926 21:33:14.567729 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.567246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50526)
0926 21:33:14.567447 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.567544 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 21:33:14.568727 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.568398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50542)
0926 21:33:14.568544 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.568621 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 21:33:14.568977 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
W20260926 21:33:14.641661 13953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:14.641969 13953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:14.642138 13953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:14.649616 13953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:14.649727 13953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:14.649824 13953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:14.649907 13953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:14.649994 13953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:14.655143 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 4
I20260926 21:33:14.656469 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.655893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50546)
0926 21:33:14.656154 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.656267 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 21:33:14.657608 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.657221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50554)
0926 21:33:14.657382 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.657470 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 21:33:14.657838 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
I20260926 21:33:14.661152 13953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44507,127.0.97.61:40499,127.0.97.60:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:14.662595 13953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:14.664012 13953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 21:33:14.675204 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 3
I20260926 21:33:14.676327 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.675875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50568)
0926 21:33:14.676080 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.676172 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:33:14.676990 13958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:14.677227 13961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:14.677626 13959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:14.677783 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.677195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50578)
0926 21:33:14.677505 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.677600 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260926 21:33:14.678033 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43825: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43825: connect: Connection refused (error 111)
I20260926 21:33:14.678582 13953 server_base.cc:1034] running on GCE node
I20260926 21:33:14.679482 13953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:14.680962 13953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:14.682170 13953 hybrid_clock.cc:648] HybridClock initialized: now 1790458394682121 us; error 44 us; skew 500 ppm
I20260926 21:33:14.682622 13953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:14.685809 13953 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 21:33:14.687441 13953 webserver.cc:460] Webserver started at http://127.0.97.60:39371/ using document root <none> and password file <none>
I20260926 21:33:14.688162 13953 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:14.688314 13953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:14.688619 13953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:14.691751 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/data/instance:
uuid: "ec0941c1b1f14f0baf32e4c48856ca0f"
format_stamp: "Formatted at 2026-09-26 21:33:14 on dist-test-slave-87bt"
server_key: "980221d8f359a28c627e00cedfb4ff51"
server_key_iv: "48897ba8830223d632fec38a7a80f4bb"
server_key_version: "encryptionkey@0"
I20260926 21:33:14.692546 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/wal/instance:
uuid: "ec0941c1b1f14f0baf32e4c48856ca0f"
format_stamp: "Formatted at 2026-09-26 21:33:14 on dist-test-slave-87bt"
server_key: "980221d8f359a28c627e00cedfb4ff51"
server_key_iv: "48897ba8830223d632fec38a7a80f4bb"
server_key_version: "encryptionkey@0"
I20260926 21:33:14.697970 13953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:33:14.701448 13968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:14.702872 13953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:33:14.703052 13953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/wal
uuid: "ec0941c1b1f14f0baf32e4c48856ca0f"
format_stamp: "Formatted at 2026-09-26 21:33:14 on dist-test-slave-87bt"
server_key: "980221d8f359a28c627e00cedfb4ff51"
server_key_iv: "48897ba8830223d632fec38a7a80f4bb"
server_key_version: "encryptionkey@0"
I20260926 21:33:14.703284 13953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:14.725579 13953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:14.729717 13953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:14.730015 13953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:14.748886 13953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43825
I20260926 21:33:14.748919 14019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43825 every 8 connection(s)
I20260926 21:33:14.750281 13953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:33:14.750598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 13953
I20260926 21:33:14.750813 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458187148965-388-0/minicluster-data/master-2/wal/instance
I20260926 21:33:14.751327 388 external_mini_cluster.cc:1428] Setting key b2280bf2d97388a648542ae4f59ed57b
I20260926 21:33:14.754747 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:14.763592 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:14.777268 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.753376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.0.97.60:43825)
0926 21:33:14.754148 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.754159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.754200 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:33:14.756467 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.756501 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.756890 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.757226 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.759033 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.760633 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.762552 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.763203 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.763339 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.768457 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.768468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.768480 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.768532 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.772489 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.773404 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.773411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.773418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.773607 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.774033 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.774043 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.774048 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.774816 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.775136 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.775531 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.776017 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":376,"threads_started":1}
I20260926 21:33:14.783900 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.767439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.0.97.62:44507)
0926 21:33:14.767642 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.767649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.767872 (+ 223us) server_negotiation.cc:407] Connection header received
0926 21:33:14.768055 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.768063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.768151 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.768264 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.769928 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.770850 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.772900 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.773517 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.773608 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.777498 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.777504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.777507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.777540 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.780715 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.781753 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.781758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.781762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.781920 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.782305 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.782312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.782315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.782631 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.782821 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.783274 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.783499 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:33:14.783926 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.766845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44507 (local address 127.0.0.1:58788)
0926 21:33:14.767720 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.767764 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:14.767925 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:14.768358 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:14.768365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:14.769352 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:14.769765 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.769782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.771006 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.771012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.772767 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.772777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.773699 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.773705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.773868 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.774644 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:14.774666 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:14.777257 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:14.780869 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.780880 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.780905 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.781627 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.782040 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.782044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.782048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.782191 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.782774 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:14.782783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:14.782970 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 21:33:14.783157 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 21:33:14.783500 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":477,"threads_started":1}
I20260926 21:33:14.787119 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:33:14.801968 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.780679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.0.97.62:44507)
0926 21:33:14.784449 (+ 3770us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.784454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.784470 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:33:14.784512 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.784516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.784689 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.784787 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.786222 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.787474 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.789003 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.789418 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.789548 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.795477 (+ 5929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.795482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.795485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.795523 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.798947 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.799743 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.799773 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.799777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.799928 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.800384 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.800390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.800396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.800796 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.801087 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.801309 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.801545 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3433,"thread_start_us":13919,"threads_started":1}
I20260926 21:33:14.805858 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.787851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40499 (local address 127.0.0.1:53662)
0926 21:33:14.788053 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.788072 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:14.788172 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:14.791594 (+ 3422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:14.791603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:14.792138 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:14.792358 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.792372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.793625 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.793629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.794786 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.794806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.795879 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.795886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.796133 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.796669 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:14.796694 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:14.799293 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:14.803168 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.803180 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.803183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.803731 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.804290 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.804295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.804299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.804460 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.805147 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:14.805151 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:14.805283 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 21:33:14.805400 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 21:33:14.805611 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 21:33:14.806550 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.790810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.0.97.61:40499)
0926 21:33:14.791049 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.791056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.791081 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:33:14.791145 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.791153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.791242 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.791366 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.792552 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.793452 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.795017 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.795621 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.795723 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.799489 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.799494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.799497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.799531 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.803024 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.803924 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.803931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.803934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.804136 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.804577 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.804584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.804587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.805021 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.805213 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.805596 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.805878 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 21:33:14.807551 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:33:14.821034 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.803927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.0.97.61:40499)
0926 21:33:14.804523 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.804527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.804545 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:33:14.804583 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.804587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.804643 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.804724 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.806337 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.807479 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.809328 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.809790 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.809920 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.814940 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.814946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.814949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.815013 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.818148 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.818988 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.818996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.819005 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.819133 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.819529 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.819534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.819536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.819821 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.820101 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.820412 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.820642 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":498,"thread_start_us":300,"threads_started":1}
I20260926 21:33:14.821640 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.808372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50592)
0926 21:33:14.808634 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.808652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:14.808759 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:14.809028 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:14.809033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:14.809490 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:14.809620 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.809628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.810711 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.810716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.812223 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.812241 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.812909 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.812915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.813134 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.813786 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:14.813804 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:14.816150 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:14.819261 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.819266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.819270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.819621 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.819984 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.819990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.819993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.820112 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.820587 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:14.820591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:14.820729 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 21:33:14.820875 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 21:33:14.821264 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 21:33:14.822484 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.808293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.0.97.60:43825)
0926 21:33:14.808458 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.808463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.808717 (+ 254us) server_negotiation.cc:407] Connection header received
0926 21:33:14.808849 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.808852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.808903 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.808976 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.809751 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.810598 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.812360 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.812751 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.812838 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.816339 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.816344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.816349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.816385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.819114 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.819747 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.819754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.819757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.819876 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.820209 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.820213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.820219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.820479 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.820624 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.821932 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.822179 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:33:14.826354 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:33:14.827962 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: Bootstrap starting.
I20260926 21:33:14.832163 14020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:14.833884 14020 log.cc:824] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:14.847632 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: No bootstrap required, opened a new log
I20260926 21:33:14.855266 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 4
I20260926 21:33:14.859529 14020 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
I20260926 21:33:14.860131 14020 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:14.860297 14020 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0941c1b1f14f0baf32e4c48856ca0f, State: Initialized, Role: FOLLOWER
I20260926 21:33:14.861210 14020 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
I20260926 21:33:14.861940 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.831896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.0.97.62:44507)
0926 21:33:14.832085 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.832090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.843075 (+ 10985us) server_negotiation.cc:407] Connection header received
0926 21:33:14.844023 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.844029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.844126 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.844208 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.846917 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.847664 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.849750 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.850188 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.850351 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.854367 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.854375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.854380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.854459 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.858297 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.859305 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.859312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.859316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.859487 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.860015 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.860024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.860029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.860411 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.860677 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.861289 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.861550 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:33:14.863919 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.841815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.0.97.61:40499)
0926 21:33:14.842075 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.842085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.845668 (+ 3583us) server_negotiation.cc:407] Connection header received
0926 21:33:14.845824 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.845831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.845923 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.846038 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.848261 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.849355 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.851451 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.852125 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.852278 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.856201 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.856211 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.856217 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.856277 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.859909 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.861188 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.861194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.861197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.861361 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.861927 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.861935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.861939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.862303 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.862599 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.863279 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.863518 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 21:33:14.869176 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:14.869665 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:33:14.871865 14020 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:14.878515 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.831530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.0.97.60:43825)
0926 21:33:14.831735 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.831743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.844664 (+ 12921us) server_negotiation.cc:407] Connection header received
0926 21:33:14.844899 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.844905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.844979 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.845083 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.853485 (+ 8402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.854694 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.866405 (+ 11711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.867135 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.867279 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.871716 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.871723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.871730 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.871769 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.875713 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.876466 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.876471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.876476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.876612 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.877004 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.877009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.877013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.877309 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.877561 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.877902 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.878128 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:33:14.879384 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.856565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.0.97.60:43825)
0926 21:33:14.862352 (+ 5787us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.862358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.862385 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:33:14.862469 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.862475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.862551 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.862667 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.863778 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.864920 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.866797 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.867467 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.867602 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.870951 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.870956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.870959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.870989 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.875627 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.876832 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.876841 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.876847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.877035 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.877591 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.877599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.877605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.878095 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.878262 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.878971 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.879123 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5653,"thread_start_us":5451,"threads_started":1}
I20260926 21:33:14.879817 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.855851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50616)
0926 21:33:14.856234 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.856253 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:14.856356 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:14.862716 (+ 6360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:14.862723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:14.863316 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:14.863599 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.863616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.865090 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.865096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.866609 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.866630 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.867621 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.867627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.867817 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.868535 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:14.868552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:14.870733 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:14.875889 (+ 5156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.875898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.875905 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.876627 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.877210 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.877218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.877224 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.877432 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.878361 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:14.878370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:14.878625 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 21:33:14.878875 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 21:33:14.879284 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260926 21:33:14.885535 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08: Bootstrap starting.
I20260926 21:33:14.889722 13872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08: Neither blocks nor log segments found. Creating new log.
W20260926 21:33:14.890753 14047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:33:14.890885 14047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:33:14.891080 13872 log.cc:824] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:14.894570 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08: No bootstrap required, opened a new log
I20260926 21:33:14.886307 13953 master_runner.cc:186] Error getting master registration for 127.0.97.62:44507: OK, instance_id { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" instance_seqno: 1790458393710554 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:33:14.897420 13953 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:33:14.905763 13872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
I20260926 21:33:14.906226 13872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:14.906356 13872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a375cdcd8484fa7b49eebab781f0a08, State: Initialized, Role: FOLLOWER
I20260926 21:33:14.907100 13872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
I20260926 21:33:14.908156 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:14.908505 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:33:14.909564 13872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:33:14.921751 14061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:33:14.921907 14061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:33:14.920624 13805 master_runner.cc:186] Error getting master registration for 127.0.97.61:40499: OK, instance_id { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" instance_seqno: 1790458394224315 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:33:14.932224 13805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:33:14.945361 14050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:33:14.945698 14050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
I20260926 21:33:14.947563 14050 leader_election.cc:288] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c083a24c24c94cdd987ab961b485d582 (127.0.97.61:40499), ec0941c1b1f14f0baf32e4c48856ca0f (127.0.97.60:43825)
I20260926 21:33:14.948652 13995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a375cdcd8484fa7b49eebab781f0a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" is_pre_election: true
I20260926 21:33:14.949313 13995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a375cdcd8484fa7b49eebab781f0a08 in term 0.
W20260926 21:33:14.949834 13826 tablet.cc:2369] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:33:14.950084 13824 leader_election.cc:302] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a375cdcd8484fa7b49eebab781f0a08, ec0941c1b1f14f0baf32e4c48856ca0f; no voters:
I20260926 21:33:14.950584 14050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:33:14.950734 14050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:33:14.950866 14050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:14.955667 14050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
I20260926 21:33:14.957052 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } attempt: 5
I20260926 21:33:14.957443 13995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a375cdcd8484fa7b49eebab781f0a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f"
I20260926 21:33:14.957693 13995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:14.961365 13995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a375cdcd8484fa7b49eebab781f0a08 in term 1.
I20260926 21:33:14.962386 13824 leader_election.cc:302] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a375cdcd8484fa7b49eebab781f0a08, ec0941c1b1f14f0baf32e4c48856ca0f; no voters:
I20260926 21:33:14.967080 14050 leader_election.cc:288] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [CANDIDATE]: Term 1 election: Requested vote from peers c083a24c24c94cdd987ab961b485d582 (127.0.97.61:40499), ec0941c1b1f14f0baf32e4c48856ca0f (127.0.97.60:43825)
I20260926 21:33:14.967342 14050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:14.967495 14050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [term 1 LEADER]: Becoming Leader. State: Replica: 4a375cdcd8484fa7b49eebab781f0a08, State: Running, Role: LEADER
I20260926 21:33:14.968153 14050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
Sep 26 21:33:14 dist-test-slave-87bt krb5kdc[13792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260926 21:33:14.982252 13974 tablet.cc:2369] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:33:14.987946 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a375cdcd8484fa7b49eebab781f0a08. Latest consensus state: current_term: 1 leader_uuid: "4a375cdcd8484fa7b49eebab781f0a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:14.988313 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:14.990605 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.957609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43825 (local address 127.0.0.1:50618)
0926 21:33:14.957871 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:14.957893 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:14.958007 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:14.962161 (+ 4154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:14.962170 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:14.962824 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:14.963056 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.963072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.964482 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.964489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.966131 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:14.966148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.971882 (+ 5734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.971890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:14.972396 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.973219 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:14.973249 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:14.976206 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:14.984397 (+ 8191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.984405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.984410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.985055 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.986284 (+ 1229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:14.986291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:14.986295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:14.986598 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:14.988167 (+ 1569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:14.988173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:14.988745 (+ 572us) client_negotiation.cc:769] Sending connection context
0926 21:33:14.989023 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 21:33:14.989689 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 21:33:14.991508 14065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:33:14.993425 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:14.957837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.0.97.60:43825)
0926 21:33:14.961745 (+ 3908us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:14.961752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:14.961782 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:33:14.961866 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:14.961871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:14.961983 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:14.962097 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:14.963210 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.964336 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.966376 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:14.967185 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:14.967322 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:14.976520 (+ 9198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:14.976529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:14.976536 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:14.976584 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:14.983868 (+ 7284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.985425 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.985432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.985472 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.986009 (+ 537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:14.986841 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:14.986885 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:14.986889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:14.987881 (+ 992us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:14.991440 (+ 3559us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:14.991887 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 21:33:14.992516 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3781}
I20260926 21:33:14.993821 14065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:33:15.000408 13946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582: Bootstrap starting.
I20260926 21:33:15.011852 13946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:15.012602 13995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [term 1 FOLLOWER]: Refusing update from remote peer 4a375cdcd8484fa7b49eebab781f0a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:15.013610 14050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:15.015182 13946 log.cc:824] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:15.038973 13946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582: No bootstrap required, opened a new log
I20260926 21:33:15.040345 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a375cdcd8484fa7b49eebab781f0a08. Latest consensus state: current_term: 1 leader_uuid: "4a375cdcd8484fa7b49eebab781f0a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:15.040681 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:33:15.044929 14066 mvcc.cc:205] Tried to move back new op lower bound from 7333717585957969920 to 7333717585805402112. Current Snapshot: MvccSnapshot[applied={T|T < 7333717585957969920}]
I20260926 21:33:15.050998 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a375cdcd8484fa7b49eebab781f0a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:15.051512 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:15.053437 14067 mvcc.cc:205] Tried to move back new op lower bound from 7333717585957969920 to 7333717585805402112. Current Snapshot: MvccSnapshot[applied={T|T < 7333717585957969920}]
I20260926 21:33:15.054177 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a375cdcd8484fa7b49eebab781f0a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:15.054513 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:33:15.057134 14065 catalog_manager.cc:1347] Generated new cluster ID: fad62e1ab52142cbb76848c421ec5130
I20260926 21:33:15.057272 14065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:33:15.064441 13946 raft_consensus.cc:348] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
I20260926 21:33:15.065143 13946 raft_consensus.cc:374] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:15.065323 13946 raft_consensus.cc:729] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c083a24c24c94cdd987ab961b485d582, State: Initialized, Role: FOLLOWER
I20260926 21:33:15.066262 13946 consensus_queue.cc:260] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } }
I20260926 21:33:15.078979 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:15.079461 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:33:15.082063 13946 sys_catalog.cc:564] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:15.082624 13919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:15.083592 14065 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:33:15.083415 13921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a375cdcd8484fa7b49eebab781f0a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c083a24c24c94cdd987ab961b485d582" is_pre_election: true
I20260926 21:33:15.082095 13920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a375cdcd8484fa7b49eebab781f0a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c083a24c24c94cdd987ab961b485d582"
I20260926 21:33:15.085632 14065 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:33:15.086896 13919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [term 1 FOLLOWER]: Refusing update from remote peer 4a375cdcd8484fa7b49eebab781f0a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:15.087818 14050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:15.101529 14065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08: Generated new TSK 0
I20260926 21:33:15.103173 14065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:33:15.131011 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a375cdcd8484fa7b49eebab781f0a08. Latest consensus state: current_term: 1 leader_uuid: "4a375cdcd8484fa7b49eebab781f0a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:15.131528 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:33:15.132128 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a375cdcd8484fa7b49eebab781f0a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:15.132544 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a375cdcd8484fa7b49eebab781f0a08 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:15.133426 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a375cdcd8484fa7b49eebab781f0a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a375cdcd8484fa7b49eebab781f0a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44507 } } peers { permanent_uuid: "c083a24c24c94cdd987ab961b485d582" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40499 } } peers { permanent_uuid: "ec0941c1b1f14f0baf32e4c48856ca0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43825 } } }
I20260926 21:33:15.135411 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:33:15.190208 14091 catalog_manager.cc:1259] Loaded cluster ID: fad62e1ab52142cbb76848c421ec5130
I20260926 21:33:15.190439 14091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582: loading cluster ID for follower catalog manager: success
I20260926 21:33:15.194573 14091 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582: acquiring CA information for follower catalog manager: success
I20260926 21:33:15.197561 14091 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c083a24c24c94cdd987ab961b485d582: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:33:15.892911 14047 catalog_manager.cc:1259] Loaded cluster ID: fad62e1ab52142cbb76848c421ec5130
I20260926 21:33:15.893021 14047 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: loading cluster ID for follower catalog manager: success
I20260926 21:33:15.895524 14047 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: acquiring CA information for follower catalog manager: success
I20260926 21:33:15.897567 14047 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec0941c1b1f14f0baf32e4c48856ca0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:33:16.222587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 13805
I20260926 21:33:16.224576 13867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:16.338380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 13805
I20260926 21:33:16.359982 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 13879
I20260926 21:33:16.361155 13941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:16.470326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 13879
I20260926 21:33:16.490912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 13953
I20260926 21:33:16.491986 14015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:16.600939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 13953
2026-09-26T21:33:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4704 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:33:16 dist-test-slave-87bt krb5kdc[14099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:33:16 dist-test-slave-87bt krb5kdc[14099](info): set up 2 sockets
Sep 26 21:33:16 dist-test-slave-87bt krb5kdc[14099](info): commencing operation
krb5kdc: starting...
W20260926 21:33:17.710922 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:33:17 dist-test-slave-87bt krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:33:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:33:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:33:17.955600 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36547
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:18.292605 14112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:18.292922 14112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:18.293013 14112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:18.300230 14112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:18.300341 14112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:18.300387 14112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:18.300444 14112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:18.300482 14112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:18.310542 14112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:18.311880 14112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:18.313145 14112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:18.325963 14118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:18.326015 14120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:18.326404 14117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:18.327148 14112 server_base.cc:1034] running on GCE node
I20260926 21:33:18.328104 14112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:18.329696 14112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:18.330971 14112 hybrid_clock.cc:648] HybridClock initialized: now 1790458398330892 us; error 68 us; skew 500 ppm
I20260926 21:33:18.331384 14112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:18 dist-test-slave-87bt krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:18.334981 14112 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:33:18.336907 14112 webserver.cc:460] Webserver started at http://127.0.97.62:35163/ using document root <none> and password file <none>
I20260926 21:33:18.337693 14112 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:18.337833 14112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:18.338177 14112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:18.341521 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "2bc0a6ffb4064e8e9c9740dfd5641837"
format_stamp: "Formatted at 2026-09-26 21:33:18 on dist-test-slave-87bt"
server_key: "1490e1f68117149e4179b9fae68833c0"
server_key_iv: "4ae90f92189b5949a7119e94e9b12ef6"
server_key_version: "encryptionkey@0"
I20260926 21:33:18.342375 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "2bc0a6ffb4064e8e9c9740dfd5641837"
format_stamp: "Formatted at 2026-09-26 21:33:18 on dist-test-slave-87bt"
server_key: "1490e1f68117149e4179b9fae68833c0"
server_key_iv: "4ae90f92189b5949a7119e94e9b12ef6"
server_key_version: "encryptionkey@0"
I20260926 21:33:18.348383 14112 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 21:33:18.352473 14127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:18.354085 14112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:33:18.354346 14112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "2bc0a6ffb4064e8e9c9740dfd5641837"
format_stamp: "Formatted at 2026-09-26 21:33:18 on dist-test-slave-87bt"
server_key: "1490e1f68117149e4179b9fae68833c0"
server_key_iv: "4ae90f92189b5949a7119e94e9b12ef6"
server_key_version: "encryptionkey@0"
I20260926 21:33:18.354660 14112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:18.393844 14112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:18.398289 14112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:18.398689 14112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:18.418417 14112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36547
I20260926 21:33:18.418514 14178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36547 every 8 connection(s)
I20260926 21:33:18.420001 14112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:33:18.423928 14179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:18.429239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 14112
I20260926 21:33:18.429505 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:33:18.430159 388 external_mini_cluster.cc:1428] Setting key 3ebacbdcab3d3eb46b5393d0cca219ea
I20260926 21:33:18.435672 14179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837: Bootstrap starting.
I20260926 21:33:18.439664 14179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:18.441004 14179 log.cc:824] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:18.444784 14179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837: No bootstrap required, opened a new log
Sep 26 21:33:18 dist-test-slave-87bt krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:18.455943 14179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc0a6ffb4064e8e9c9740dfd5641837" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36547 } }
I20260926 21:33:18.456475 14179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:18.456597 14179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc0a6ffb4064e8e9c9740dfd5641837, State: Initialized, Role: FOLLOWER
I20260926 21:33:18.456909 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:18.432094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.0.97.62:36547)
0926 21:33:18.432951 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:18.432963 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:18.433015 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:33:18.434834 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:18.434861 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:18.435297 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:18.435672 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:18.437571 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:18.438959 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:18.440909 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:18.441869 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:18.442090 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:18.446911 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:18.446922 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:18.446942 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:18.447003 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:18.451792 (+ 4789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:18.452662 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:18.452672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:18.452682 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:18.452903 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:18.453286 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:18.453292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:18.453296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:18.454150 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:18.454486 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:18.454930 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 21:33:18.455393 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":387,"threads_started":1}
I20260926 21:33:18.457322 14179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc0a6ffb4064e8e9c9740dfd5641837" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36547 } }
I20260926 21:33:18.457597 14179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:33:18.457736 14179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:33:18.457928 14179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:18.460830 14179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc0a6ffb4064e8e9c9740dfd5641837" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36547 } }
I20260926 21:33:18.461153 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:33:18.461452 14179 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc0a6ffb4064e8e9c9740dfd5641837; no voters:
I20260926 21:33:18.462086 14179 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:33:18.462244 14184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:18.462919 14184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [term 1 LEADER]: Becoming Leader. State: Replica: 2bc0a6ffb4064e8e9c9740dfd5641837, State: Running, Role: LEADER
I20260926 21:33:18.463536 14184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc0a6ffb4064e8e9c9740dfd5641837" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36547 } }
I20260926 21:33:18.465999 14179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:18.466933 14185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc0a6ffb4064e8e9c9740dfd5641837. Latest consensus state: current_term: 1 leader_uuid: "2bc0a6ffb4064e8e9c9740dfd5641837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc0a6ffb4064e8e9c9740dfd5641837" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36547 } } }
I20260926 21:33:18.466977 14186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bc0a6ffb4064e8e9c9740dfd5641837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc0a6ffb4064e8e9c9740dfd5641837" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36547 } } }
I20260926 21:33:18.467408 14185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:18.467461 14186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:18.469476 14189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:33:18.474882 14189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:33:18.484177 14189 catalog_manager.cc:1347] Generated new cluster ID: fc1b62b9f1f9458bbb4b98e5900c7358
I20260926 21:33:18.484323 14189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:33:18.509084 14189 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:33:18.510530 14189 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:33:18.519843 14189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bc0a6ffb4064e8e9c9740dfd5641837: Generated new TSK 0
I20260926 21:33:18.520750 14189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:33:18.575197 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 14112
I20260926 21:33:18.576478 14174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:18.674283 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 14112
2026-09-26T21:33:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2069 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:33:18 dist-test-slave-87bt krb5kdc[14207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:33:18 dist-test-slave-87bt krb5kdc[14207](info): set up 2 sockets
Sep 26 21:33:18 dist-test-slave-87bt krb5kdc[14207](info): commencing operation
krb5kdc: starting...
W20260926 21:33:19.774932 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:33:19 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:33:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:33:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:33:20.013840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:20.354015 14220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:20.354342 14220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:20.354498 14220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:20.362025 14220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:20.362154 14220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:20.362202 14220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:20.362252 14220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:20.362298 14220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:20.372529 14220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:20.373911 14220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:20.375373 14220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:20.387472 14226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:20.387611 14228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:20.387472 14225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:20.388389 14220 server_base.cc:1034] running on GCE node
I20260926 21:33:20.389151 14220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:20.390589 14220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:20.391811 14220 hybrid_clock.cc:648] HybridClock initialized: now 1790458400391744 us; error 58 us; skew 500 ppm
I20260926 21:33:20.392179 14220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:20 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:20.395406 14220 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:33:20.397028 14220 webserver.cc:460] Webserver started at http://127.0.97.62:39691/ using document root <none> and password file <none>
I20260926 21:33:20.397764 14220 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:20.397890 14220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:20.398231 14220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:20.401425 14220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "ea13ee710ca24d0ea0388aed191c8022"
format_stamp: "Formatted at 2026-09-26 21:33:20 on dist-test-slave-87bt"
server_key: "2d72f5477c2fb365aeb993164e978b3f"
server_key_iv: "6221ea0cd4e8ca2414d6654c6e8423b1"
server_key_version: "encryptionkey@0"
I20260926 21:33:20.402276 14220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "ea13ee710ca24d0ea0388aed191c8022"
format_stamp: "Formatted at 2026-09-26 21:33:20 on dist-test-slave-87bt"
server_key: "2d72f5477c2fb365aeb993164e978b3f"
server_key_iv: "6221ea0cd4e8ca2414d6654c6e8423b1"
server_key_version: "encryptionkey@0"
I20260926 21:33:20.407744 14220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:33:20.411574 14235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:20.412981 14220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:33:20.413189 14220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "ea13ee710ca24d0ea0388aed191c8022"
format_stamp: "Formatted at 2026-09-26 21:33:20 on dist-test-slave-87bt"
server_key: "2d72f5477c2fb365aeb993164e978b3f"
server_key_iv: "6221ea0cd4e8ca2414d6654c6e8423b1"
server_key_version: "encryptionkey@0"
I20260926 21:33:20.413376 14220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:20.439841 14220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:20.444510 14220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:20.444772 14220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:20.465889 14220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37687
I20260926 21:33:20.465929 14286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37687 every 8 connection(s)
I20260926 21:33:20.467418 14220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:33:20.471621 14287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:20.477591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 14220
I20260926 21:33:20.477862 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:33:20.478701 388 external_mini_cluster.cc:1428] Setting key 0758df6d5605994f8493b93c64bda115
I20260926 21:33:20.483330 14287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Bootstrap starting.
I20260926 21:33:20.487214 14287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:20.488420 14287 log.cc:824] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:20.491617 14287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: No bootstrap required, opened a new log
Sep 26 21:33:20 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:20.499730 14287 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:20.500129 14287 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:20.500262 14287 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea13ee710ca24d0ea0388aed191c8022, State: Initialized, Role: FOLLOWER
I20260926 21:33:20.500949 14287 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:20.501204 14287 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:33:20.501361 14287 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:33:20.501614 14287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:20.502236 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:20.480970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.0.97.62:37687)
0926 21:33:20.481828 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:20.481840 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:20.481882 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:33:20.483711 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:20.483735 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:20.484157 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:20.484493 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:20.486351 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:20.487812 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:20.489589 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:20.490276 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:20.490515 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:20.494617 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:20.494626 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:20.494642 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:20.494687 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:20.497802 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:20.498585 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:20.498593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:20.498599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:20.498778 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:20.499180 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:20.499185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:20.499189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:20.499736 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:20.500058 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:20.500469 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 21:33:20.501009 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":407,"threads_started":1}
I20260926 21:33:20.504657 14287 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:20.505287 14287 leader_election.cc:302] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea13ee710ca24d0ea0388aed191c8022; no voters:
I20260926 21:33:20.505929 14287 leader_election.cc:288] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:33:20.506111 14292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:20.506852 14292 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 1 LEADER]: Becoming Leader. State: Replica: ea13ee710ca24d0ea0388aed191c8022, State: Running, Role: LEADER
I20260926 21:33:20.507405 14292 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:20.508162 14287 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:20.510815 14294 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea13ee710ca24d0ea0388aed191c8022. Latest consensus state: current_term: 1 leader_uuid: "ea13ee710ca24d0ea0388aed191c8022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260926 21:33:20.511528 14294 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:20.510846 14293 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea13ee710ca24d0ea0388aed191c8022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260926 21:33:20.511966 14293 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:20.513226 14301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:33:20.519327 14301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:33:20.530699 14301 catalog_manager.cc:1347] Generated new cluster ID: c1939f2808f5474aa556805933fcd8fe
I20260926 21:33:20.530839 14301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:33:20.545424 14301 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:33:20.546902 14301 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:33:20.553901 14301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Generated new TSK 0
I20260926 21:33:20.554857 14301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:33:20.610795 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:20.964028 14313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:20.964558 14313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:20.964710 14313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:20.972460 14313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:20.972620 14313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:20.972823 14313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:33:20.982956 14313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:20.984465 14313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:20.985841 14313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:21.001634 14318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:21.001734 14319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:21.002315 14321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:21.003041 14313 server_base.cc:1034] running on GCE node
I20260926 21:33:21.003795 14313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:21.006974 14313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:21.008311 14313 hybrid_clock.cc:648] HybridClock initialized: now 1790458401008158 us; error 123 us; skew 500 ppm
I20260926 21:33:21.008766 14313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:21 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:21.013139 14313 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:33:21.015239 14313 webserver.cc:460] Webserver started at http://127.0.97.1:45959/ using document root <none> and password file <none>
I20260926 21:33:21.016330 14313 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:21.016489 14313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:21.016897 14313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:21.021003 14313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "99c9c0336f1a02313c8d0b3599677584"
server_key_iv: "3c1c5d10c73b2507f6e391f0ae641337"
server_key_version: "encryptionkey@0"
I20260926 21:33:21.021865 14313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "99c9c0336f1a02313c8d0b3599677584"
server_key_iv: "3c1c5d10c73b2507f6e391f0ae641337"
server_key_version: "encryptionkey@0"
I20260926 21:33:21.027601 14313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 21:33:21.031368 14328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:21.032918 14313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:33:21.033147 14313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "99c9c0336f1a02313c8d0b3599677584"
server_key_iv: "3c1c5d10c73b2507f6e391f0ae641337"
server_key_version: "encryptionkey@0"
I20260926 21:33:21.033366 14313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:21.057657 14313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:21.062125 14313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:21.062491 14313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:21.063560 14313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:21.065056 14313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:21.065138 14313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:21.065287 14313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:21.065348 14313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:21.098913 14313 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35045
I20260926 21:33:21.098965 14440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35045 every 8 connection(s)
I20260926 21:33:21.100242 14313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:33:21.105558 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 14313
I20260926 21:33:21.105788 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:33:21.106489 388 external_mini_cluster.cc:1428] Setting key b3e3ea194530281b16a7211fb34d5fae
Sep 26 21:33:21 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:21.122632 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:21.102834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35745 (local address 127.0.97.62:37687)
0926 21:33:21.103327 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:21.103334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:21.104365 (+ 1031us) server_negotiation.cc:407] Connection header received
0926 21:33:21.106296 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:21.106304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:21.106406 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:21.106566 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:21.108321 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:21.109259 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:21.111121 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:21.111872 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:21.112037 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:21.116420 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:21.116426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:21.116429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:21.116481 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:21.119308 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:21.120151 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:21.120158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:21.120163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:21.120291 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:21.120751 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:21.120757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:21.120760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:21.121040 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:21.121210 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:21.121965 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 21:33:21.122221 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260926 21:33:21.123646 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:21.103396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:35745)
0926 21:33:21.104191 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:21.104238 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:21.106006 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:21.106827 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:21.106836 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:21.107460 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:21.108085 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:21.108104 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:21.109479 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:21.109483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:21.110942 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:21.110955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:21.112172 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:21.112180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:21.112545 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:21.113417 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:21.113451 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:21.116143 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:21.119488 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:21.119497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:21.119525 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:21.119995 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:21.120410 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:21.120415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:21.120419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:21.120631 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:21.121183 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:21.121191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:21.121559 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 21:33:21.121852 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 21:33:21.122472 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":340,"threads_started":1}
I20260926 21:33:21.125391 14441 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260926 21:33:21.125706 14441 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:21.126466 14441 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260926 21:33:21.128595 14252 ts_manager.cc:194] Registered new tserver with Master: c3e0f581d0ae4000a0d4d29f7fb158f8 (127.0.97.1:35045)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:33:21.130183 14252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:33:21.158155 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:21.495940 14447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:21.496309 14447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:21.496485 14447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:21.503897 14447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:21.504086 14447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:21.504276 14447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:33:21.515223 14447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:21.516601 14447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:21.518038 14447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:21.531909 14452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:21.531975 14453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:21.531949 14455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:21.533073 14447 server_base.cc:1034] running on GCE node
I20260926 21:33:21.533716 14447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:21.535020 14447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:21.536309 14447 hybrid_clock.cc:648] HybridClock initialized: now 1790458401536269 us; error 72 us; skew 500 ppm
I20260926 21:33:21.536666 14447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:21 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:21.540328 14447 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:33:21.542026 14447 webserver.cc:460] Webserver started at http://127.0.97.2:37889/ using document root <none> and password file <none>
I20260926 21:33:21.542860 14447 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:21.543030 14447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:21.543355 14447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:21.546828 14447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "85db6b506ee041c5840520b540019d6e"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "e1f6c1aa702fcf35fb98d123385f9156"
server_key_iv: "4ddb5a5df434c9dac62c7456104167c1"
server_key_version: "encryptionkey@0"
I20260926 21:33:21.547706 14447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "85db6b506ee041c5840520b540019d6e"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "e1f6c1aa702fcf35fb98d123385f9156"
server_key_iv: "4ddb5a5df434c9dac62c7456104167c1"
server_key_version: "encryptionkey@0"
I20260926 21:33:21.553299 14447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:33:21.557193 14462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:21.558725 14447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:33:21.558907 14447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "85db6b506ee041c5840520b540019d6e"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "e1f6c1aa702fcf35fb98d123385f9156"
server_key_iv: "4ddb5a5df434c9dac62c7456104167c1"
server_key_version: "encryptionkey@0"
I20260926 21:33:21.559134 14447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:21.596058 14447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:21.600642 14447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:21.600939 14447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:21.601958 14447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:21.603317 14447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:21.603420 14447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:21.603580 14447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:21.603654 14447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:21.637040 14447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35501
I20260926 21:33:21.637089 14574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35501 every 8 connection(s)
I20260926 21:33:21.638491 14447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:33:21.642339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 14447
I20260926 21:33:21.642545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:33:21.643127 388 external_mini_cluster.cc:1428] Setting key cbdceb805a05e51fd1b2fb091275bb7c
Sep 26 21:33:21 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:21.660029 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:21.641111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57075 (local address 127.0.97.62:37687)
0926 21:33:21.641559 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:21.641564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:21.642569 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 21:33:21.644873 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:21.644877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:21.644972 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:21.645089 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:21.646746 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:21.647919 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:21.649616 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:21.650390 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:21.650563 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:21.654167 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:21.654172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:21.654176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:21.654208 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:21.656843 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:21.657629 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:21.657641 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:21.657643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:21.657766 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:21.658209 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:21.658215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:21.658218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:21.658543 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:21.658714 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:21.659419 (+ 705us) server_negotiation.cc:299] Negotiation successful
0926 21:33:21.659653 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":238,"threads_started":1}
I20260926 21:33:21.661015 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:21.641541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:57075)
0926 21:33:21.642360 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:21.642436 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:21.644573 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:21.645399 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:21.645408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:21.645921 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:21.646562 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:21.646582 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:21.648079 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:21.648083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:21.649447 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:21.649460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:21.650630 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:21.650635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:21.650833 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:21.651556 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:21.651583 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:21.653911 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:21.657022 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:21.657031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:21.657050 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:21.657490 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:21.657883 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:21.657888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:21.657892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:21.658088 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:21.658687 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:21.658695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:21.659052 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 21:33:21.659334 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 21:33:21.659885 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":307,"threads_started":1}
I20260926 21:33:21.662514 14575 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260926 21:33:21.662861 14575 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:21.663843 14575 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260926 21:33:21.665450 14252 ts_manager.cc:194] Registered new tserver with Master: 85db6b506ee041c5840520b540019d6e (127.0.97.2:35501)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:33:21.666312 14252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:33:21.695196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:22.027599 14581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:22.027901 14581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:22.027997 14581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:22.035923 14581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:22.036044 14581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:22.036167 14581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:33:22.046162 14581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:22.047518 14581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:22.048842 14581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:22.062538 14586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:22.062821 14589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:22.062945 14587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:22.063817 14581 server_base.cc:1034] running on GCE node
I20260926 21:33:22.064487 14581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:22.065555 14581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:22.066794 14581 hybrid_clock.cc:648] HybridClock initialized: now 1790458402066755 us; error 47 us; skew 500 ppm
I20260926 21:33:22.067148 14581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:22 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:22.070472 14581 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:33:22.071992 14581 webserver.cc:460] Webserver started at http://127.0.97.3:36515/ using document root <none> and password file <none>
I20260926 21:33:22.072713 14581 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:22.072861 14581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:22.073181 14581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:22.076197 14581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "637ba4b09bd244ff824d0ba01fdb069b"
format_stamp: "Formatted at 2026-09-26 21:33:22 on dist-test-slave-87bt"
server_key: "62b49ea9f0b9364d453b6f16c9210f9c"
server_key_iv: "bfe70370507a1af202981513e717f6e5"
server_key_version: "encryptionkey@0"
I20260926 21:33:22.076982 14581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "637ba4b09bd244ff824d0ba01fdb069b"
format_stamp: "Formatted at 2026-09-26 21:33:22 on dist-test-slave-87bt"
server_key: "62b49ea9f0b9364d453b6f16c9210f9c"
server_key_iv: "bfe70370507a1af202981513e717f6e5"
server_key_version: "encryptionkey@0"
I20260926 21:33:22.082144 14581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:33:22.085724 14596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:22.087138 14581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 21:33:22.087319 14581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "637ba4b09bd244ff824d0ba01fdb069b"
format_stamp: "Formatted at 2026-09-26 21:33:22 on dist-test-slave-87bt"
server_key: "62b49ea9f0b9364d453b6f16c9210f9c"
server_key_iv: "bfe70370507a1af202981513e717f6e5"
server_key_version: "encryptionkey@0"
I20260926 21:33:22.087531 14581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:22.126116 14581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:22.130865 14581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:22.131191 14581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:22.132273 14581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:22.133674 14581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:22.133776 14581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:22.133927 14581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:22.134002 14581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:22.134323 14441 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260926 21:33:22.168900 14581 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37731
I20260926 21:33:22.169059 14708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37731 every 8 connection(s)
I20260926 21:33:22.170215 14581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:33:22.179594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 14581
I20260926 21:33:22.179809 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:33:22.180365 388 external_mini_cluster.cc:1428] Setting key 489eb483da931c676f11453ce30b25b6
Sep 26 21:33:22 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:22.191094 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.173019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39791 (local address 127.0.97.62:37687)
0926 21:33:22.173509 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.173515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.174495 (+ 980us) server_negotiation.cc:407] Connection header received
0926 21:33:22.176410 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.176417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.176528 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.176651 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:22.178167 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.179024 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.181256 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.181814 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.181956 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.185691 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:22.185695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:22.185699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:22.185732 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:22.188259 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.188951 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.188954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.188956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.189076 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.189433 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.189436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.189438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.189685 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:22.189792 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.190488 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.190725 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":237,"threads_started":1}
I20260926 21:33:22.192065 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.173614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:39791)
0926 21:33:22.174312 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.174349 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:22.176137 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:22.176889 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:22.176898 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:22.177417 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:22.177991 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.178010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.179184 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.179188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.181107 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.181120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.182030 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.182035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.182293 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.183062 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:22.183084 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:22.185467 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:22.188396 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:22.188403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:22.188416 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:22.188845 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:22.189172 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:22.189175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:22.189177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:22.189345 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:22.189792 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:22.189797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:22.190106 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 21:33:22.190344 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 21:33:22.190839 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":280,"threads_started":1}
I20260926 21:33:22.193557 14709 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260926 21:33:22.193941 14709 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:22.194926 14709 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260926 21:33:22.196551 14252 ts_manager.cc:194] Registered new tserver with Master: 637ba4b09bd244ff824d0ba01fdb069b (127.0.97.3:37731)
I20260926 21:33:22.197392 14252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39791
I20260926 21:33:22.208851 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:33:22.226030 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.213507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.97.62:37687)
0926 21:33:22.213695 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.213699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.214233 (+ 534us) server_negotiation.cc:407] Connection header received
0926 21:33:22.214354 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.214358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.214493 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.214583 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:22.215929 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.216745 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.218188 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.218663 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.218762 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.221042 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:22.221047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:22.221050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:22.221086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:22.223776 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.224381 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.224385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.224388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.224510 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.224848 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.224851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.224853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.225161 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:22.225339 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.225606 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.225801 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:33:22.230293 14252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:33:22.237632 14252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:33:22.273231 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.250395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36444)
0926 21:33:22.252201 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.252222 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:22.252338 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:22.264902 (+ 12564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:22.264914 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:22.264954 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:22.265394 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.265408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.267573 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.267580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.270592 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.270612 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.272436 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.272441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.272611 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.272640 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 21:33:22.272714 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 21:33:22.272816 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1642,"mutex_wait_us":141,"thread_start_us":883,"threads_started":1}
I20260926 21:33:22.273262 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.252368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.0.97.3:37731)
0926 21:33:22.265192 (+ 12824us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.265200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.265232 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:33:22.265331 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.265342 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.265607 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.265816 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:22.266546 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.268097 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.270366 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.271890 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.272040 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.272305 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.272710 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.272892 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12646,"thread_start_us":12469,"threads_started":1}
I20260926 21:33:22.273976 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.251230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.0.97.1:35045)
0926 21:33:22.264015 (+ 12785us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.264029 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.264087 (+ 58us) server_negotiation.cc:407] Connection header received
0926 21:33:22.264236 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.264249 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.264593 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.265667 (+ 1074us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:22.265917 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.267407 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.270762 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.272219 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.272359 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.272586 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.272974 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.273176 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12547,"thread_start_us":12377,"threads_started":1}
I20260926 21:33:22.274564 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.251723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37731 (local address 127.0.0.1:45672)
0926 21:33:22.263501 (+ 11778us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.263953 (+ 452us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:22.264148 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:22.265895 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:22.265901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:22.265930 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:22.266335 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.266346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.268269 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.268274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.270179 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.270193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.272089 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.272094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.272332 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.272400 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 21:33:22.272541 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 21:33:22.272765 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11588,"mutex_wait_us":2,"thread_start_us":11425,"threads_started":1}
I20260926 21:33:22.275148 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.250098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35501 (local address 127.0.0.1:34314)
0926 21:33:22.251509 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.251556 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:22.251772 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:22.265392 (+ 13620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:22.265397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:22.265423 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:22.265841 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.265858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.267529 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.267536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.270612 (+ 3076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.270621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.274246 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.274254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.274467 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.274509 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 21:33:22.274585 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 21:33:22.274731 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":162,"thread_start_us":716,"threads_started":1}
I20260926 21:33:22.276206 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.250522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.97.2:35501)
0926 21:33:22.264794 (+ 14272us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.264807 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.264844 (+ 37us) server_negotiation.cc:407] Connection header received
0926 21:33:22.264943 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.264950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.265195 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.265402 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:22.266058 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.267366 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.272420 (+ 5054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.273977 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.275543 (+ 1566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.275763 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.275836 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.275965 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14049,"thread_start_us":13895,"threads_started":1}
I20260926 21:33:22.278339 14376 tablet_service.cc:1467] Processing CreateTablet for tablet ebb7c89520664290a6b42084347e7662 (DEFAULT_TABLE table=test-table [id=a0f4c005d938443b9d046e8d9cd55b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:22.278339 14644 tablet_service.cc:1467] Processing CreateTablet for tablet ebb7c89520664290a6b42084347e7662 (DEFAULT_TABLE table=test-table [id=a0f4c005d938443b9d046e8d9cd55b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:22.279119 14510 tablet_service.cc:1467] Processing CreateTablet for tablet ebb7c89520664290a6b42084347e7662 (DEFAULT_TABLE table=test-table [id=a0f4c005d938443b9d046e8d9cd55b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:22.279906 14644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb7c89520664290a6b42084347e7662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:22.279906 14376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb7c89520664290a6b42084347e7662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:22.280393 14510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb7c89520664290a6b42084347e7662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:22.291968 14728 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Bootstrap starting.
I20260926 21:33:22.294555 14730 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Bootstrap starting.
I20260926 21:33:22.295341 14729 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Bootstrap starting.
I20260926 21:33:22.295823 14728 tablet_bootstrap.cc:654] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:22.297178 14728 log.cc:824] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:22.298882 14729 tablet_bootstrap.cc:654] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:22.299877 14728 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: No bootstrap required, opened a new log
I20260926 21:33:22.300158 14728 ts_tablet_manager.cc:1397] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:33:22.300241 14729 log.cc:824] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:22.301512 14730 tablet_bootstrap.cc:654] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:22.303157 14730 log.cc:824] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:22.306111 14730 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: No bootstrap required, opened a new log
I20260926 21:33:22.306506 14730 ts_tablet_manager.cc:1397] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 21:33:22.306665 14729 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: No bootstrap required, opened a new log
I20260926 21:33:22.306931 14729 ts_tablet_manager.cc:1397] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260926 21:33:22.310025 14728 raft_consensus.cc:348] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:22.310536 14728 raft_consensus.cc:374] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:22.310654 14728 raft_consensus.cc:729] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85db6b506ee041c5840520b540019d6e, State: Initialized, Role: FOLLOWER
I20260926 21:33:22.311321 14728 consensus_queue.cc:260] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:22.312566 14575 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260926 21:33:22.313234 14728 ts_tablet_manager.cc:1428] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 21:33:22.316053 14730 raft_consensus.cc:348] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:22.316444 14730 raft_consensus.cc:374] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:22.316561 14730 raft_consensus.cc:729] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e0f581d0ae4000a0d4d29f7fb158f8, State: Initialized, Role: FOLLOWER
I20260926 21:33:22.317224 14730 consensus_queue.cc:260] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:22.317276 14729 raft_consensus.cc:348] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:22.317756 14729 raft_consensus.cc:374] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:22.317915 14729 raft_consensus.cc:729] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637ba4b09bd244ff824d0ba01fdb069b, State: Initialized, Role: FOLLOWER
I20260926 21:33:22.319307 14730 ts_tablet_manager.cc:1428] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 21:33:22.319945 14729 consensus_queue.cc:260] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:22.321079 14709 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260926 21:33:22.321700 14729 ts_tablet_manager.cc:1428] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 21:33:22.351289 14735 raft_consensus.cc:482] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:33:22.351598 14735 raft_consensus.cc:504] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
W20260926 21:33:22.352665 14442 tablet.cc:2369] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:33:22.353444 14735 leader_election.cc:288] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85db6b506ee041c5840520b540019d6e (127.0.97.2:35501), 637ba4b09bd244ff824d0ba01fdb069b (127.0.97.3:37731)
I20260926 21:33:22.359995 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.353998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55425 (local address 127.0.97.3:37731)
0926 21:33:22.354340 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.354348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.354754 (+ 406us) server_negotiation.cc:407] Connection header received
0926 21:33:22.355005 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.355009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.355067 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.355169 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:22.355695 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.356403 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.358202 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.359037 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.359191 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.359300 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.359583 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.359723 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 21:33:22.360203 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.353705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37731 (local address 127.0.97.1:55425)
0926 21:33:22.354630 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.354648 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:22.354844 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:22.355187 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:22.355191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:22.355216 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:22.355540 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.355550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.356531 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.356539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.358036 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.358048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.359248 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.359254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.359420 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.359448 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 21:33:22.359556 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 21:33:22.359712 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":3,"thread_start_us":563,"threads_started":1}
I20260926 21:33:22.360205 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.353879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57277 (local address 127.0.97.2:35501)
0926 21:33:22.354120 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.354127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.355037 (+ 910us) server_negotiation.cc:407] Connection header received
0926 21:33:22.355240 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.355244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.355302 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.355399 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:22.355946 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.356648 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.358352 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.359233 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.359394 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.359496 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.359801 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.359931 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:33:22.360522 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.353705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35501 (local address 127.0.97.1:57277)
0926 21:33:22.354928 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.354942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:22.355100 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:22.355415 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:22.355420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:22.355448 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:22.355799 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.355807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.356787 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.356793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.358197 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:22.358208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.359457 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.359463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:22.359634 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.359656 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:33:22.359766 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 21:33:22.359907 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":96,"thread_start_us":781,"threads_started":1}
I20260926 21:33:22.361495 14664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb7c89520664290a6b42084347e7662" candidate_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "637ba4b09bd244ff824d0ba01fdb069b" is_pre_election: true
I20260926 21:33:22.361624 14530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb7c89520664290a6b42084347e7662" candidate_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85db6b506ee041c5840520b540019d6e" is_pre_election: true
I20260926 21:33:22.362056 14664 raft_consensus.cc:2454] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e0f581d0ae4000a0d4d29f7fb158f8 in term 0.
I20260926 21:33:22.362118 14530 raft_consensus.cc:2454] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e0f581d0ae4000a0d4d29f7fb158f8 in term 0.
I20260926 21:33:22.362847 14332 leader_election.cc:302] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637ba4b09bd244ff824d0ba01fdb069b, c3e0f581d0ae4000a0d4d29f7fb158f8; no voters:
I20260926 21:33:22.363313 14735 raft_consensus.cc:2781] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:33:22.363438 14735 raft_consensus.cc:482] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:33:22.363508 14735 raft_consensus.cc:3037] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:22.365615 14735 raft_consensus.cc:504] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:22.366250 14735 leader_election.cc:288] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [CANDIDATE]: Term 1 election: Requested vote from peers 85db6b506ee041c5840520b540019d6e (127.0.97.2:35501), 637ba4b09bd244ff824d0ba01fdb069b (127.0.97.3:37731)
I20260926 21:33:22.366694 14530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb7c89520664290a6b42084347e7662" candidate_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85db6b506ee041c5840520b540019d6e"
I20260926 21:33:22.366919 14530 raft_consensus.cc:3037] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:22.366863 14664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb7c89520664290a6b42084347e7662" candidate_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "637ba4b09bd244ff824d0ba01fdb069b"
I20260926 21:33:22.367101 14664 raft_consensus.cc:3037] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:22.369189 14530 raft_consensus.cc:2454] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e0f581d0ae4000a0d4d29f7fb158f8 in term 1.
I20260926 21:33:22.369310 14664 raft_consensus.cc:2454] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e0f581d0ae4000a0d4d29f7fb158f8 in term 1.
I20260926 21:33:22.369611 14330 leader_election.cc:302] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85db6b506ee041c5840520b540019d6e, c3e0f581d0ae4000a0d4d29f7fb158f8; no voters:
I20260926 21:33:22.369927 14735 raft_consensus.cc:2781] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:22.370378 14735 raft_consensus.cc:686] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 LEADER]: Becoming Leader. State: Replica: c3e0f581d0ae4000a0d4d29f7fb158f8, State: Running, Role: LEADER
I20260926 21:33:22.370956 14735 consensus_queue.cc:237] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:22.375703 14251 catalog_manager.cc:5495] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e0f581d0ae4000a0d4d29f7fb158f8 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } health_report { overall_health: UNKNOWN } } }
W20260926 21:33:22.390808 14576 tablet.cc:2369] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:33:22.417948 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.412475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.0.97.1:35045)
0926 21:33:22.412692 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.412698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.412719 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:33:22.412819 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.412823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.412897 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.413056 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:33:22.413434 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.414148 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.415675 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.416093 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.416242 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.416800 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:33:22.417365 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:33:22.417536 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.417586 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.417669 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 21:33:22.422060 14710 tablet.cc:2369] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:33:22.426580 14664 raft_consensus.cc:1261] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 1 FOLLOWER]: Refusing update from remote peer c3e0f581d0ae4000a0d4d29f7fb158f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:22.426592 14530 raft_consensus.cc:1261] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 1 FOLLOWER]: Refusing update from remote peer c3e0f581d0ae4000a0d4d29f7fb158f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:22.427831 14739 consensus_queue.cc:1035] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:22.428224 14735 consensus_queue.cc:1035] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:22.458222 14252 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:33:22.458604 14252 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:33:22.462260 14252 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Sending DeleteTablet for 3 replicas of tablet ebb7c89520664290a6b42084347e7662
I20260926 21:33:22.463775 14376 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb7c89520664290a6b42084347e7662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36444
I20260926 21:33:22.464123 14510 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb7c89520664290a6b42084347e7662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34314
I20260926 21:33:22.464262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 14313
I20260926 21:33:22.467084 14644 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb7c89520664290a6b42084347e7662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45672
I20260926 21:33:22.480827 14753 tablet_replica.cc:330] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: stopping tablet replica
I20260926 21:33:22.481320 14753 raft_consensus.cc:2229] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:22.481855 14753 raft_consensus.cc:2258] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:22.484134 14755 tablet_replica.cc:330] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: stopping tablet replica
I20260926 21:33:22.484537 14755 raft_consensus.cc:2229] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:22.485024 14755 raft_consensus.cc:2258] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:22.487287 14753 ts_tablet_manager.cc:1905] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:33:22 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:33:22.493644 14755 ts_tablet_manager.cc:1905] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:22.499402 14752 tablet_replica.cc:330] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: stopping tablet replica
I20260926 21:33:22.499879 14752 raft_consensus.cc:2229] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:33:22.500397 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.465437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.0.97.1:35045)
0926 21:33:22.465649 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.465654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.476599 (+ 10945us) server_negotiation.cc:407] Connection header received
0926 21:33:22.477911 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.477921 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.478035 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.478147 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:22.482259 (+ 4112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.483572 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.485720 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.486485 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.486582 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.492458 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:22.492475 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:22.492493 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:22.492549 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:22.496628 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.497616 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.497632 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.497638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.497815 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.498323 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.498330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.498335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.498833 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:22.499126 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.499751 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.500008 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 21:33:22.500669 14752 raft_consensus.cc:2258] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:22.500808 14755 ts_tablet_manager.cc:1918] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:22.500984 14755 log.cc:1196] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/ebb7c89520664290a6b42084347e7662
I20260926 21:33:22.501683 14755 ts_tablet_manager.cc:1939] T ebb7c89520664290a6b42084347e7662 P 637ba4b09bd244ff824d0ba01fdb069b: Deleting consensus metadata
I20260926 21:33:22.501778 14436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:22.503906 14239 catalog_manager.cc:4841] TS 637ba4b09bd244ff824d0ba01fdb069b (127.0.97.3:37731): tablet ebb7c89520664290a6b42084347e7662 (table test-table [id=a0f4c005d938443b9d046e8d9cd55b06]) successfully deleted
I20260926 21:33:22.507043 14753 ts_tablet_manager.cc:1918] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:22.507233 14753 log.cc:1196] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/ebb7c89520664290a6b42084347e7662
I20260926 21:33:22.507762 14753 ts_tablet_manager.cc:1939] T ebb7c89520664290a6b42084347e7662 P 85db6b506ee041c5840520b540019d6e: Deleting consensus metadata
I20260926 21:33:22.509409 14237 catalog_manager.cc:4841] TS 85db6b506ee041c5840520b540019d6e (127.0.97.2:35501): tablet ebb7c89520664290a6b42084347e7662 (table test-table [id=a0f4c005d938443b9d046e8d9cd55b06]) successfully deleted
I20260926 21:33:22.602001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 14313
W20260926 21:33:22.616961 14239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35045 (error 108)
I20260926 21:33:22.617717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 14447
I20260926 21:33:22.619146 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.618546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36464)
0926 21:33:22.618790 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.618948 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260926 21:33:22.619421 14239 catalog_manager.cc:4584] TS c3e0f581d0ae4000a0d4d29f7fb158f8 (127.0.97.1:35045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebb7c89520664290a6b42084347e7662: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35045: connect: Connection refused (error 111)
Sep 26 21:33:22 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:33:22.633697 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.618615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.0.97.2:35501)
0926 21:33:22.618796 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.618800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.618818 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:33:22.618942 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.618945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.619005 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.619097 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:22.620426 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.621174 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.622498 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.622909 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.623012 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.627899 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:22.627916 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:22.627932 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:22.627975 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:22.630922 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.631681 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.631700 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.631702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.631840 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.632227 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.632232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.632234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.632660 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:22.632904 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.633215 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.633420 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:33:22.634517 14570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:22.673007 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.672522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36468)
0926 21:33:22.672764 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.672850 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 21:33:22.674034 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.673695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36472)
0926 21:33:22.673838 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.673914 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 21:33:22.727008 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 14447
I20260926 21:33:22.734680 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.734170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36474)
0926 21:33:22.734381 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.734510 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 21:33:22.735740 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.735385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36486)
0926 21:33:22.735536 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.735621 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 21:33:22.743902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 14581
Sep 26 21:33:22 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:33:22.759799 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.744732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.0.97.3:37731)
0926 21:33:22.744897 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:22.744902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:22.745006 (+ 104us) server_negotiation.cc:407] Connection header received
0926 21:33:22.745134 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:22.745138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:22.745206 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:22.745294 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:22.746671 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.747407 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.748857 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:22.749275 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:22.749376 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:22.754006 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:22.754023 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:22.754039 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:22.754083 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:22.757026 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.757795 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.757803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.757805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.757942 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:22.758325 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:22.758329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:22.758331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:22.758833 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:22.759063 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:22.759334 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 21:33:22.759511 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:33:22.760483 14704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:22.816226 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.815773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36498)
0926 21:33:22.815979 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.816074 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 21:33:22.817366 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:22.816993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36500)
0926 21:33:22.817193 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:22.817259 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 21:33:22.855370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 14581
I20260926 21:33:22.871734 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 14220
I20260926 21:33:22.872923 14282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:22.983031 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 14220
I20260926 21:33:23.004258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--webserver_interface=127.0.97.62
--webserver_port=39691
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:23.344933 14761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:23.345268 14761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:23.345427 14761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:23.353135 14761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:23.353279 14761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:23.353369 14761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:23.353461 14761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:23.353531 14761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:23.364163 14761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:23.365598 14761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:23.367127 14761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:23.390282 14768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:23.390301 14770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:23.390548 14767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:23.391297 14761 server_base.cc:1034] running on GCE node
I20260926 21:33:23.392225 14761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:23.393862 14761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:23.395099 14761 hybrid_clock.cc:648] HybridClock initialized: now 1790458403395034 us; error 47 us; skew 500 ppm
I20260926 21:33:23.395598 14761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:23 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:23.404038 14761 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:33:23.408574 14761 webserver.cc:460] Webserver started at http://127.0.97.62:39691/ using document root <none> and password file <none>
I20260926 21:33:23.409754 14761 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:23.409942 14761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:23.419766 14761 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 21:33:23.424130 14777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:23.426191 14761 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:33:23.426466 14761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "ea13ee710ca24d0ea0388aed191c8022"
format_stamp: "Formatted at 2026-09-26 21:33:20 on dist-test-slave-87bt"
server_key: "2d72f5477c2fb365aeb993164e978b3f"
server_key_iv: "6221ea0cd4e8ca2414d6654c6e8423b1"
server_key_version: "encryptionkey@0"
I20260926 21:33:23.428200 14761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:23.457836 14761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:23.463349 14761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:23.463662 14761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:23.484886 14761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37687
I20260926 21:33:23.484896 14828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37687 every 8 connection(s)
I20260926 21:33:23.486555 14761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:33:23.489090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 14761
I20260926 21:33:23.489912 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35045
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=45959
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:33:23.493775 14829 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:33:23.498039 14829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Bootstrap starting.
I20260926 21:33:23.528422 14829 log.cc:824] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:23.544340 14829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:33:23.545084 14829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Bootstrap complete.
I20260926 21:33:23.560822 14829 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:23.562103 14829 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea13ee710ca24d0ea0388aed191c8022, State: Initialized, Role: FOLLOWER
I20260926 21:33:23.562958 14829 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:23.563271 14829 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:33:23.563408 14829 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:33:23.563596 14829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:33:23.569334 14829 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:23.569977 14829 leader_election.cc:302] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea13ee710ca24d0ea0388aed191c8022; no voters:
I20260926 21:33:23.570709 14829 leader_election.cc:288] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 21:33:23.570879 14833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:33:23.572311 14833 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 2 LEADER]: Becoming Leader. State: Replica: ea13ee710ca24d0ea0388aed191c8022, State: Running, Role: LEADER
I20260926 21:33:23.573004 14829 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:23.572841 14833 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:23.575309 14835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea13ee710ca24d0ea0388aed191c8022. Latest consensus state: current_term: 2 leader_uuid: "ea13ee710ca24d0ea0388aed191c8022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260926 21:33:23.575733 14835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:23.576258 14834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea13ee710ca24d0ea0388aed191c8022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260926 21:33:23.576630 14834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:23.578855 14839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:33:23.589000 14839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:33:23.590448 14839 catalog_manager.cc:1259] Loaded cluster ID: c1939f2808f5474aa556805933fcd8fe
I20260926 21:33:23.590575 14839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:33:23.593808 14839 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 21:33:23.611508 14839 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 21:33:23.634060 14839 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:33:23.636296 14839 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Loaded TSK: 0
I20260926 21:33:23.637953 14839 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 21:33:23.880200 14831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:23.880496 14831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:23.880633 14831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:23.888242 14831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:23.888365 14831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:23.888538 14831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:33:23.898659 14831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45959
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:23.899993 14831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:23.901345 14831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:23.917992 14857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:23.917904 14858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:23.918859 14860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:23.919486 14831 server_base.cc:1034] running on GCE node
I20260926 21:33:23.920205 14831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:23.921274 14831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:23.922564 14831 hybrid_clock.cc:648] HybridClock initialized: now 1790458403922460 us; error 68 us; skew 500 ppm
I20260926 21:33:23.922923 14831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:23 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:23.926405 14831 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:33:23.928409 14831 webserver.cc:460] Webserver started at http://127.0.97.1:45959/ using document root <none> and password file <none>
I20260926 21:33:23.929188 14831 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:23.929364 14831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:23.935575 14831 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 21:33:23.938877 14867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:23.940510 14831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:33:23.940701 14831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "99c9c0336f1a02313c8d0b3599677584"
server_key_iv: "3c1c5d10c73b2507f6e391f0ae641337"
server_key_version: "encryptionkey@0"
I20260926 21:33:23.941751 14831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:23.969946 14831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:23.975173 14831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:23.975502 14831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:23.976660 14831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:23.978802 14874 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 21:33:23.984649 14831 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 21:33:23.984813 14831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 21:33:23.984964 14831 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 21:33:23.988229 14831 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 21:33:23.988341 14831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260926 21:33:23.988682 14874 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Bootstrap starting.
I20260926 21:33:23.999804 14874 log.cc:824] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:24.020417 14874 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:33:24.021448 14874 tablet_bootstrap.cc:492] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Bootstrap complete.
I20260926 21:33:24.022177 14874 ts_tablet_manager.cc:1397] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.008s
I20260926 21:33:24.035001 14831 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35045
I20260926 21:33:24.035049 14981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35045 every 8 connection(s)
I20260926 21:33:24.036778 14831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:33:24.036420 14874 raft_consensus.cc:348] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:24.038093 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 14831
I20260926 21:33:24.038081 14874 raft_consensus.cc:729] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e0f581d0ae4000a0d4d29f7fb158f8, State: Initialized, Role: FOLLOWER
I20260926 21:33:24.038887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35501
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=37889
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:33:24.039143 14874 consensus_queue.cc:260] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } }
I20260926 21:33:24.041584 14874 ts_tablet_manager.cc:1428] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
Sep 26 21:33:24 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:24.070237 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:24.043693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:51607)
0926 21:33:24.047137 (+ 3444us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:24.047187 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:24.047778 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:24.049538 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:24.049547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:24.050192 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:24.050935 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:24.050958 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.053595 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.053599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:24.055026 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:24.055042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.056676 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.056681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:24.056921 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:24.057693 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:24.057718 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:24.060209 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:24.065751 (+ 5542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:24.065760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:24.065774 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:24.066341 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:24.066943 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:24.066947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:24.066949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:24.067137 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:24.068104 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:24.068110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:24.068524 (+ 414us) client_negotiation.cc:769] Sending connection context
0926 21:33:24.068811 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 21:33:24.069221 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3107,"thread_start_us":2859,"threads_started":1}
I20260926 21:33:24.070739 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:24.043753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51607 (local address 127.0.97.62:37687)
0926 21:33:24.048006 (+ 4253us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:24.048024 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:24.048076 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:33:24.048471 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:24.048492 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:24.049004 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:24.049424 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:24.051210 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.053385 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.055214 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.056451 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.056676 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:24.060620 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:24.060636 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:24.060662 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:24.060745 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:24.065486 (+ 4741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:24.066555 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:24.066566 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:24.066575 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:24.066823 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:24.067260 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:24.067266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:24.067269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:24.067967 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:24.068170 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:24.069089 (+ 919us) server_negotiation.cc:299] Negotiation successful
0926 21:33:24.069555 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4018,"thread_start_us":3877,"threads_started":1}
I20260926 21:33:24.073691 14982 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260926 21:33:24.074056 14982 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:24.074947 14982 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260926 21:33:24.078830 14794 ts_manager.cc:194] Registered new tserver with Master: c3e0f581d0ae4000a0d4d29f7fb158f8 (127.0.97.1:35045)
I20260926 21:33:24.083513 14794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51607
I20260926 21:33:24.088373 14982 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260926 21:33:24.097817 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:24.082557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.0.97.1:35045)
0926 21:33:24.091679 (+ 9122us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:24.091686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:24.091712 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:33:24.091805 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:24.091810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:24.091979 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:24.092194 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:24.092819 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.093888 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.095687 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.096601 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.096754 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:24.097000 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:24.097263 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 21:33:24.097420 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8947,"thread_start_us":8781,"threads_started":1}
I20260926 21:33:24.097792 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:24.082301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.0.1:36516)
0926 21:33:24.090118 (+ 7817us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:24.090181 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:24.090370 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:24.092210 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:24.092217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:24.092249 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:24.092619 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:24.092629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.094043 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.094050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:24.095511 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:24.095525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.096785 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.096791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:24.097021 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:24.097077 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 21:33:24.097223 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 21:33:24.097425 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7542,"thread_start_us":8327,"threads_started":1}
I20260926 21:33:24.099332 14917 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb7c89520664290a6b42084347e7662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36516
I20260926 21:33:24.101513 14990 tablet_replica.cc:330] stopping tablet replica
I20260926 21:33:24.102049 14990 raft_consensus.cc:2229] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:24.102615 14990 raft_consensus.cc:2258] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:24.104321 14990 ts_tablet_manager.cc:1905] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:24.114534 14990 ts_tablet_manager.cc:1918] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:24.114722 14990 log.cc:1196] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/ebb7c89520664290a6b42084347e7662
I20260926 21:33:24.115258 14990 ts_tablet_manager.cc:1939] T ebb7c89520664290a6b42084347e7662 P c3e0f581d0ae4000a0d4d29f7fb158f8: Deleting consensus metadata
I20260926 21:33:24.117796 14781 catalog_manager.cc:4841] TS c3e0f581d0ae4000a0d4d29f7fb158f8 (127.0.97.1:35045): tablet ebb7c89520664290a6b42084347e7662 (table test-table [id=a0f4c005d938443b9d046e8d9cd55b06]) successfully deleted
W20260926 21:33:24.392016 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:24.392319 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:24.392483 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:24.399901 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:24.400033 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:24.400230 14987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:33:24.410254 14987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37889
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:24.411753 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:24.413172 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:24.427723 14996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:24.427721 14995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:24.427719 14998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:24.428812 14987 server_base.cc:1034] running on GCE node
I20260926 21:33:24.429584 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:24.430784 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:24.432025 14987 hybrid_clock.cc:648] HybridClock initialized: now 1790458404431966 us; error 59 us; skew 500 ppm
I20260926 21:33:24.432368 14987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:24 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:24.436259 14987 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:33:24.437974 14987 webserver.cc:460] Webserver started at http://127.0.97.2:37889/ using document root <none> and password file <none>
I20260926 21:33:24.438839 14987 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:24.439011 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:24.444832 14987 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:33:24.448020 15005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:24.449478 14987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:33:24.449672 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "85db6b506ee041c5840520b540019d6e"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "e1f6c1aa702fcf35fb98d123385f9156"
server_key_iv: "4ddb5a5df434c9dac62c7456104167c1"
server_key_version: "encryptionkey@0"
I20260926 21:33:24.450857 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:24.480520 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:24.485383 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:24.485699 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:24.486809 14987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:24.488204 14987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:24.488327 14987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:24.488464 14987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:24.488559 14987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:24.523406 14987 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35501
I20260926 21:33:24.523442 15117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35501 every 8 connection(s)
I20260926 21:33:24.524926 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:33:24.526957 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 14987
I20260926 21:33:24.527678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37731
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=36515
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:33:24 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:24.548291 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:24.527891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37571 (local address 127.0.97.62:37687)
0926 21:33:24.528097 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:24.528103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:24.529499 (+ 1396us) server_negotiation.cc:407] Connection header received
0926 21:33:24.531543 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:24.531552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:24.531664 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:24.531803 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:24.533325 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.534231 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.536935 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.537391 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.537502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:24.541911 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:24.541916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:24.541919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:24.541955 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:24.544757 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:24.545666 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:24.545671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:24.545673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:24.545831 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:24.546394 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:24.546398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:24.546400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:24.546724 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:24.546891 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:24.547664 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 21:33:24.547914 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 21:33:24.549335 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:24.528441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:37571)
0926 21:33:24.529303 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:24.529351 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:24.531237 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:24.532015 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:24.532024 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:24.532551 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:24.533131 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:24.533148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.534394 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.534403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:24.536754 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:24.536773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:24.537648 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:24.537658 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:24.538018 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:24.538861 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:24.538896 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:24.541638 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:24.544983 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:24.544994 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:24.545013 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:24.545500 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:24.545975 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:24.545980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:24.545984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:24.546257 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:24.546866 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:24.546874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:24.547269 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 21:33:24.547555 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 21:33:24.548146 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":361,"threads_started":1}
I20260926 21:33:24.551030 15118 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260926 21:33:24.551424 15118 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:24.552440 15118 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260926 21:33:24.554179 14793 ts_manager.cc:194] Registered new tserver with Master: 85db6b506ee041c5840520b540019d6e (127.0.97.2:35501)
I20260926 21:33:24.555503 14793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37571
W20260926 21:33:24.871158 15121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:24.871452 15121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:24.871552 15121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:24.879056 15121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:24.879181 15121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:24.879297 15121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:33:24.889010 15121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36515
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:24.890285 15121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:24.891569 15121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:24.905345 15126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:24.905445 15127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:24.905408 15129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:24.906212 15121 server_base.cc:1034] running on GCE node
I20260926 21:33:24.906986 15121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:24.907998 15121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:24.909232 15121 hybrid_clock.cc:648] HybridClock initialized: now 1790458404909184 us; error 54 us; skew 500 ppm
I20260926 21:33:24.909559 15121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:24 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:24.912935 15121 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:33:24.914381 15121 webserver.cc:460] Webserver started at http://127.0.97.3:36515/ using document root <none> and password file <none>
I20260926 21:33:24.915126 15121 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:24.915287 15121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:24.920871 15121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 21:33:24.923614 15136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:24.924912 15121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 21:33:24.925097 15121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "637ba4b09bd244ff824d0ba01fdb069b"
format_stamp: "Formatted at 2026-09-26 21:33:22 on dist-test-slave-87bt"
server_key: "62b49ea9f0b9364d453b6f16c9210f9c"
server_key_iv: "bfe70370507a1af202981513e717f6e5"
server_key_version: "encryptionkey@0"
I20260926 21:33:24.926100 15121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:24.959277 15121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:24.963987 15121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:24.964277 15121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:24.965287 15121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:24.966810 15121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:24.966948 15121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:24.967080 15121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:24.967167 15121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:25.001025 15121 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37731
I20260926 21:33:25.001111 15248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37731 every 8 connection(s)
I20260926 21:33:25.002637 15121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:33:25.003335 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 15121
Sep 26 21:33:25 dist-test-slave-87bt krb5kdc[14207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:25.020022 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.004393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.0.97.62:37687)
0926 21:33:25.004634 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.004644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.005098 (+ 454us) server_negotiation.cc:407] Connection header received
0926 21:33:25.005306 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.005311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.005400 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.005515 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:25.007315 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.008256 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.009779 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.010305 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.010484 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.013756 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:25.013761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:25.013764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:25.013799 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:25.017221 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.017958 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.017965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.017969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.018104 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.018524 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.018530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.018533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.018814 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:25.019106 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.019425 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.019650 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:33:25.026938 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.006709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44437 (local address 127.0.97.62:37687)
0926 21:33:25.007641 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.007646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.008663 (+ 1017us) server_negotiation.cc:407] Connection header received
0926 21:33:25.010568 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.010573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.010675 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.010817 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:25.013214 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.013965 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.016563 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.017106 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.017261 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.020912 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:25.020918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:25.020923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:25.020963 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:25.023741 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.024571 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.024576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.024578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.024702 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.025145 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.025155 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.025157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.025417 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:25.025567 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.026283 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.026542 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":736,"threads_started":1}
I20260926 21:33:25.027891 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.006317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:44437)
0926 21:33:25.008516 (+ 2199us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:25.008557 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:25.010217 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:25.011150 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:25.011161 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:25.012046 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:25.012981 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.013013 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.014145 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.014151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.016389 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.016408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.017364 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.017372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.017602 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.018257 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:25.018281 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:25.020690 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:25.023923 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:25.023933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:25.023957 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:25.024428 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:25.024819 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:25.024824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:25.024827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:25.025022 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:25.025544 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:25.025552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:25.025925 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 21:33:25.026192 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 21:33:25.026781 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":471,"threads_started":1}
I20260926 21:33:25.029395 15249 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37687
I20260926 21:33:25.029660 15249 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:25.030390 15249 heartbeater.cc:511] Master 127.0.97.62:37687 requested a full tablet report, sending...
I20260926 21:33:25.031910 14793 ts_manager.cc:194] Registered new tserver with Master: 637ba4b09bd244ff824d0ba01fdb069b (127.0.97.3:37731)
I20260926 21:33:25.032722 14793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44437
I20260926 21:33:25.039295 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:33:25.047937 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.040367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.0.97.62:37687)
0926 21:33:25.040594 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.040600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.040937 (+ 337us) server_negotiation.cc:407] Connection header received
0926 21:33:25.041160 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.041163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.041317 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.041439 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:33:25.041944 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.042784 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.045040 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.045587 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.045684 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.046527 (+ 843us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:33:25.047184 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:33:25.047322 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.047472 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.047550 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:33:25.049161 14793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:33:25.056631 14793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:33:25.071020 14917 tablet_service.cc:1467] Processing CreateTablet for tablet 8f40dd24ab33438e91d39a1c3159c588 (DEFAULT_TABLE table=test-table [id=949d93e8841247c896f15655efe4f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:25.072024 14917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f40dd24ab33438e91d39a1c3159c588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:25.079244 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.068187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35501 (local address 127.0.0.1:34342)
0926 21:33:25.069110 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:25.069144 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:25.069319 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:25.072387 (+ 3068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:25.072392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:25.072422 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:25.072784 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.072803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.074710 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.074716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.076245 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.076269 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.078278 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.078285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.078513 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.078552 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 21:33:25.078627 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 21:33:25.078773 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":26,"thread_start_us":593,"threads_started":1}
I20260926 21:33:25.080112 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.068652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37731 (local address 127.0.0.1:45690)
0926 21:33:25.069871 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:25.069891 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:25.069994 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:25.072888 (+ 2894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:25.072893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:25.072923 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:25.073265 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.073277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.075968 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.075974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.077704 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.077717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.079315 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.079321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.079491 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.079521 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 21:33:25.079601 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 21:33:25.079732 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":20,"thread_start_us":957,"threads_started":1}
I20260926 21:33:25.080642 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.069405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.0.97.3:37731)
0926 21:33:25.072204 (+ 2799us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.072213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.072258 (+ 45us) server_negotiation.cc:407] Connection header received
0926 21:33:25.072379 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.072387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.072645 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.072874 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:25.073711 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.075784 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.077912 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.079065 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.079235 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.079786 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.079867 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.080035 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2624,"thread_start_us":2454,"threads_started":1}
I20260926 21:33:25.080744 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.068406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.0.97.2:35501)
0926 21:33:25.071658 (+ 3252us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.071667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.071702 (+ 35us) server_negotiation.cc:407] Connection header received
0926 21:33:25.071829 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.071837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.072151 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.072382 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:25.073010 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.074581 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.076459 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.078021 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.079686 (+ 1665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.079980 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.080071 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.080337 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3017,"spinlock_wait_cycles":25984,"thread_start_us":2879,"threads_started":1}
I20260926 21:33:25.080919 15263 tablet_bootstrap.cc:492] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: Bootstrap starting.
I20260926 21:33:25.082935 15263 tablet_bootstrap.cc:654] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:25.084698 15053 tablet_service.cc:1467] Processing CreateTablet for tablet 8f40dd24ab33438e91d39a1c3159c588 (DEFAULT_TABLE table=test-table [id=949d93e8841247c896f15655efe4f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:25.084698 15184 tablet_service.cc:1467] Processing CreateTablet for tablet 8f40dd24ab33438e91d39a1c3159c588 (DEFAULT_TABLE table=test-table [id=949d93e8841247c896f15655efe4f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:25.085376 15263 tablet_bootstrap.cc:492] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: No bootstrap required, opened a new log
I20260926 21:33:25.085541 15263 ts_tablet_manager.cc:1397] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 21:33:25.086025 15053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f40dd24ab33438e91d39a1c3159c588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:25.086091 15184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f40dd24ab33438e91d39a1c3159c588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:25.086414 15263 raft_consensus.cc:348] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.086740 15263 raft_consensus.cc:374] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:25.086836 15263 raft_consensus.cc:729] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e0f581d0ae4000a0d4d29f7fb158f8, State: Initialized, Role: FOLLOWER
I20260926 21:33:25.087080 15263 consensus_queue.cc:260] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.095067 15263 ts_tablet_manager.cc:1428] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260926 21:33:25.101698 15266 tablet_bootstrap.cc:492] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: Bootstrap starting.
I20260926 21:33:25.103055 15267 tablet_bootstrap.cc:492] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Bootstrap starting.
I20260926 21:33:25.105154 15266 tablet_bootstrap.cc:654] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:25.105813 15267 tablet_bootstrap.cc:654] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:25.106269 15266 log.cc:824] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:25.106830 15267 log.cc:824] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:25.109576 15266 tablet_bootstrap.cc:492] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: No bootstrap required, opened a new log
I20260926 21:33:25.109582 15267 tablet_bootstrap.cc:492] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: No bootstrap required, opened a new log
I20260926 21:33:25.109881 15266 ts_tablet_manager.cc:1397] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:33:25.109865 15267 ts_tablet_manager.cc:1397] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260926 21:33:25.121311 15266 raft_consensus.cc:348] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.121311 15267 raft_consensus.cc:348] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.121855 15267 raft_consensus.cc:374] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:25.121855 15266 raft_consensus.cc:374] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:25.121970 15267 raft_consensus.cc:729] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637ba4b09bd244ff824d0ba01fdb069b, State: Initialized, Role: FOLLOWER
I20260926 21:33:25.121970 15266 raft_consensus.cc:729] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85db6b506ee041c5840520b540019d6e, State: Initialized, Role: FOLLOWER
I20260926 21:33:25.122766 15267 consensus_queue.cc:260] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.122757 15266 consensus_queue.cc:260] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.124014 15118 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260926 21:33:25.124485 15249 heartbeater.cc:503] Master 127.0.97.62:37687 was elected leader, sending a full tablet report...
I20260926 21:33:25.124855 15266 ts_tablet_manager.cc:1428] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260926 21:33:25.125020 15267 ts_tablet_manager.cc:1428] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260926 21:33:25.173218 15271 raft_consensus.cc:482] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:33:25.173555 15271 raft_consensus.cc:504] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.175311 15271 leader_election.cc:288] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 637ba4b09bd244ff824d0ba01fdb069b (127.0.97.3:37731), c3e0f581d0ae4000a0d4d29f7fb158f8 (127.0.97.1:35045)
I20260926 21:33:25.182703 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.175692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49571 (local address 127.0.97.3:37731)
0926 21:33:25.175904 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.175909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.176413 (+ 504us) server_negotiation.cc:407] Connection header received
0926 21:33:25.176677 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.176684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.176767 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.176840 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:25.177659 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.178628 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.180551 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.181497 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.181600 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.181679 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.182129 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.182397 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 21:33:25.182839 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.176502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59153 (local address 127.0.97.1:35045)
0926 21:33:25.177233 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.177241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.177258 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:33:25.177327 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.177332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.177404 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.177513 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:25.178131 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.178996 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.180844 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.181712 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.181862 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.181972 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.182383 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.182561 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":449,"threads_started":1}
I20260926 21:33:25.182861 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.175576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37731 (local address 127.0.97.2:49571)
0926 21:33:25.176235 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:25.176254 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:25.176454 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:25.177083 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:25.177087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:25.177116 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:25.177502 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.177515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.178766 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.178772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.180395 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.180407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.181731 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.181739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.181942 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.181978 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 21:33:25.182074 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 21:33:25.182331 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":27,"thread_start_us":323,"threads_started":1}
I20260926 21:33:25.183317 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.176348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35045 (local address 127.0.97.2:59153)
0926 21:33:25.177045 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:25.177069 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:25.177185 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:25.177530 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:25.177533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:25.177549 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:25.177969 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.177981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.179141 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.179145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.180691 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:25.180705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.181924 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.181929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:25.182096 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.182126 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 21:33:25.182217 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 21:33:25.182486 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":54,"thread_start_us":292,"threads_started":1}
I20260926 21:33:25.183959 15204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f40dd24ab33438e91d39a1c3159c588" candidate_uuid: "85db6b506ee041c5840520b540019d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "637ba4b09bd244ff824d0ba01fdb069b" is_pre_election: true
I20260926 21:33:25.184155 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f40dd24ab33438e91d39a1c3159c588" candidate_uuid: "85db6b506ee041c5840520b540019d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" is_pre_election: true
I20260926 21:33:25.184506 15204 raft_consensus.cc:2454] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85db6b506ee041c5840520b540019d6e in term 0.
I20260926 21:33:25.184700 14937 raft_consensus.cc:2454] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85db6b506ee041c5840520b540019d6e in term 0.
I20260926 21:33:25.185189 15009 leader_election.cc:302] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637ba4b09bd244ff824d0ba01fdb069b, 85db6b506ee041c5840520b540019d6e; no voters:
I20260926 21:33:25.185550 15271 raft_consensus.cc:2781] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:33:25.185700 15271 raft_consensus.cc:482] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:33:25.185814 15271 raft_consensus.cc:3037] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:25.188090 15271 raft_consensus.cc:504] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.188705 15271 leader_election.cc:288] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [CANDIDATE]: Term 1 election: Requested vote from peers 637ba4b09bd244ff824d0ba01fdb069b (127.0.97.3:37731), c3e0f581d0ae4000a0d4d29f7fb158f8 (127.0.97.1:35045)
I20260926 21:33:25.189121 15204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f40dd24ab33438e91d39a1c3159c588" candidate_uuid: "85db6b506ee041c5840520b540019d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "637ba4b09bd244ff824d0ba01fdb069b"
I20260926 21:33:25.189204 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f40dd24ab33438e91d39a1c3159c588" candidate_uuid: "85db6b506ee041c5840520b540019d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8"
I20260926 21:33:25.189329 15204 raft_consensus.cc:3037] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:25.189464 14937 raft_consensus.cc:3037] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:25.191852 15204 raft_consensus.cc:2454] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85db6b506ee041c5840520b540019d6e in term 1.
I20260926 21:33:25.191979 14937 raft_consensus.cc:2454] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85db6b506ee041c5840520b540019d6e in term 1.
I20260926 21:33:25.192391 15009 leader_election.cc:302] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637ba4b09bd244ff824d0ba01fdb069b, 85db6b506ee041c5840520b540019d6e; no voters:
I20260926 21:33:25.192731 15271 raft_consensus.cc:2781] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:25.193212 15271 raft_consensus.cc:686] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 1 LEADER]: Becoming Leader. State: Replica: 85db6b506ee041c5840520b540019d6e, State: Running, Role: LEADER
I20260926 21:33:25.193719 15271 consensus_queue.cc:237] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } }
I20260926 21:33:25.199128 14794 catalog_manager.cc:5495] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85db6b506ee041c5840520b540019d6e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "85db6b506ee041c5840520b540019d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db6b506ee041c5840520b540019d6e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 } health_report { overall_health: UNKNOWN } } }
I20260926 21:33:25.240284 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.234658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.0.97.2:35501)
0926 21:33:25.234902 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.234907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.234926 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:33:25.235191 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.235200 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.235306 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.235442 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:33:25.235744 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.236466 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.237948 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.238347 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.238513 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.239078 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:33:25.239618 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:33:25.239733 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.239878 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.239954 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 21:33:25.249238 14937 raft_consensus.cc:1261] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Refusing update from remote peer 85db6b506ee041c5840520b540019d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:25.249243 15204 raft_consensus.cc:1261] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 1 FOLLOWER]: Refusing update from remote peer 85db6b506ee041c5840520b540019d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:25.250294 15275 consensus_queue.cc:1035] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "637ba4b09bd244ff824d0ba01fdb069b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:25.250770 15271 consensus_queue.cc:1035] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:33:25.254508 15250 tablet.cc:2369] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:33:25.267854 15280 mvcc.cc:205] Tried to move back new op lower bound from 7333717627889655808 to 7333717627677925376. Current Snapshot: MvccSnapshot[applied={T|T < 7333717627889655808}]
I20260926 21:33:25.271358 15278 mvcc.cc:205] Tried to move back new op lower bound from 7333717627889655808 to 7333717627677925376. Current Snapshot: MvccSnapshot[applied={T|T < 7333717627889655808 or (T in {7333717627889655808})}]
I20260926 21:33:25.278481 14794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:33:25.278825 14794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:33:25.282661 14794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Sending DeleteTablet for 3 replicas of tablet 8f40dd24ab33438e91d39a1c3159c588
I20260926 21:33:25.283723 14917 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f40dd24ab33438e91d39a1c3159c588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36516
I20260926 21:33:25.283872 15053 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f40dd24ab33438e91d39a1c3159c588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34342
I20260926 21:33:25.283902 15184 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f40dd24ab33438e91d39a1c3159c588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45690
I20260926 21:33:25.285068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 14831
I20260926 21:33:25.296211 15288 tablet_replica.cc:330] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: stopping tablet replica
I20260926 21:33:25.296445 15287 tablet_replica.cc:330] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: stopping tablet replica
I20260926 21:33:25.296658 15287 raft_consensus.cc:2229] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:25.296775 15288 raft_consensus.cc:2229] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:33:25.297006 15287 raft_consensus.cc:2258] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:25.297461 15288 raft_consensus.cc:2258] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:25.298594 15287 ts_tablet_manager.cc:1905] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:25.299714 15288 ts_tablet_manager.cc:1905] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:25.300643 15289 tablet_replica.cc:330] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: stopping tablet replica
I20260926 21:33:25.301076 15289 raft_consensus.cc:2229] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:25.301578 15289 raft_consensus.cc:2258] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:25.303571 15289 ts_tablet_manager.cc:1905] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:25.306867 15287 ts_tablet_manager.cc:1918] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:25.307082 15287 log.cc:1196] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/8f40dd24ab33438e91d39a1c3159c588
I20260926 21:33:25.307631 15287 ts_tablet_manager.cc:1939] T 8f40dd24ab33438e91d39a1c3159c588 P c3e0f581d0ae4000a0d4d29f7fb158f8: Deleting consensus metadata
I20260926 21:33:25.307701 15288 ts_tablet_manager.cc:1918] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:25.307890 15288 log.cc:1196] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/8f40dd24ab33438e91d39a1c3159c588
I20260926 21:33:25.308457 15288 ts_tablet_manager.cc:1939] T 8f40dd24ab33438e91d39a1c3159c588 P 85db6b506ee041c5840520b540019d6e: Deleting consensus metadata
I20260926 21:33:25.309103 14781 catalog_manager.cc:4841] TS c3e0f581d0ae4000a0d4d29f7fb158f8 (127.0.97.1:35045): tablet 8f40dd24ab33438e91d39a1c3159c588 (table test-table [id=949d93e8841247c896f15655efe4f850]) successfully deleted
I20260926 21:33:25.310240 15289 ts_tablet_manager.cc:1918] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:25.310288 14779 catalog_manager.cc:4841] TS 85db6b506ee041c5840520b540019d6e (127.0.97.2:35501): tablet 8f40dd24ab33438e91d39a1c3159c588 (table test-table [id=949d93e8841247c896f15655efe4f850]) successfully deleted
I20260926 21:33:25.310403 15289 log.cc:1196] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/8f40dd24ab33438e91d39a1c3159c588
I20260926 21:33:25.310917 15289 ts_tablet_manager.cc:1939] T 8f40dd24ab33438e91d39a1c3159c588 P 637ba4b09bd244ff824d0ba01fdb069b: Deleting consensus metadata
I20260926 21:33:25.312839 14781 catalog_manager.cc:4841] TS 637ba4b09bd244ff824d0ba01fdb069b (127.0.97.3:37731): tablet 8f40dd24ab33438e91d39a1c3159c588 (table test-table [id=949d93e8841247c896f15655efe4f850]) successfully deleted
I20260926 21:33:25.312913 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.286543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.0.97.1:35045)
0926 21:33:25.287371 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.287380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.287405 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:33:25.287483 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.287491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.287603 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.287723 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:25.290322 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.291830 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.294485 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.295409 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.295564 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.299432 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:25.299448 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:25.299465 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:25.299504 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:25.303528 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.304492 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.304501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.304504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.304684 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.310666 (+ 5982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.310672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.310676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.311209 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:25.311515 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.312192 (+ 677us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.312475 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592}
I20260926 21:33:25.313560 14977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:25.407140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 14831
I20260926 21:33:25.432112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 14987
I20260926 21:33:25.448797 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.432982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.0.97.2:35501)
0926 21:33:25.433163 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.433168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.433266 (+ 98us) server_negotiation.cc:407] Connection header received
0926 21:33:25.433453 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.433457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.433543 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.433623 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:25.435721 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.436577 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.438644 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.439097 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.439197 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.442638 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:25.442651 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:25.442665 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:25.442711 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:25.445847 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.446788 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.446795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.446798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.446924 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.447324 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.447327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.447329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.447682 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:25.447928 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.448232 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.448404 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:33:25.449640 15113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:25.569326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 14987
I20260926 21:33:25.588112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 15121
I20260926 21:33:25.601539 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:25.588923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.0.97.3:37731)
0926 21:33:25.589101 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:25.589106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:25.589168 (+ 62us) server_negotiation.cc:407] Connection header received
0926 21:33:25.589230 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:25.589233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:25.589317 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:25.589403 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:25.590755 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.591506 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.592793 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:25.593198 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:25.593298 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:25.596061 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:25.596075 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:25.596091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:25.596128 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:25.599158 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.599803 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.599810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.599812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.599957 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:25.600293 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:25.600296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:25.600298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:25.600620 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:25.600835 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:25.601096 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 21:33:25.601261 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:33:25.602192 15244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:25.690644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 15121
I20260926 21:33:25.707393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 14761
I20260926 21:33:25.708462 14824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:25.821477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 14761
I20260926 21:33:25.843241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--webserver_interface=127.0.97.62
--webserver_port=39691
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:26.186108 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:26.186440 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:26.186636 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:26.194005 15294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:26.194156 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:26.194259 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:26.194376 15294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:26.194463 15294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:26.204771 15294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:26.206357 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:26.207809 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:26.220358 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:26.220412 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:26.220436 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:26.221657 15294 server_base.cc:1034] running on GCE node
I20260926 21:33:26.222651 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:26.224273 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:26.225581 15294 hybrid_clock.cc:648] HybridClock initialized: now 1790458406225536 us; error 65 us; skew 500 ppm
I20260926 21:33:26.226001 15294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:26 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:26.229646 15294 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:33:26.231447 15294 webserver.cc:460] Webserver started at http://127.0.97.62:39691/ using document root <none> and password file <none>
I20260926 21:33:26.232203 15294 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:26.232363 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:26.238050 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 21:33:26.241060 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:26.242388 15294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 21:33:26.242599 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "ea13ee710ca24d0ea0388aed191c8022"
format_stamp: "Formatted at 2026-09-26 21:33:20 on dist-test-slave-87bt"
server_key: "2d72f5477c2fb365aeb993164e978b3f"
server_key_iv: "6221ea0cd4e8ca2414d6654c6e8423b1"
server_key_version: "encryptionkey@0"
I20260926 21:33:26.243625 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:26.267937 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:26.272652 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:26.272958 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:26.292237 15294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37687
I20260926 21:33:26.292268 15360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37687 every 8 connection(s)
I20260926 21:33:26.293800 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:33:26.298311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 15294
I20260926 21:33:26.299150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35045
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=45959
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:33:26.301082 15361 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:33:26.304508 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Bootstrap starting.
I20260926 21:33:26.325412 15361 log.cc:824] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:26.341629 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:33:26.342233 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022: Bootstrap complete.
I20260926 21:33:26.352025 15361 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:26.353024 15361 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea13ee710ca24d0ea0388aed191c8022, State: Initialized, Role: FOLLOWER
I20260926 21:33:26.353657 15361 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:26.353919 15361 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:33:26.354058 15361 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:33:26.354238 15361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 2 FOLLOWER]: Advancing to term 3
I20260926 21:33:26.358286 15361 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:26.358872 15361 leader_election.cc:302] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea13ee710ca24d0ea0388aed191c8022; no voters:
I20260926 21:33:26.359591 15361 leader_election.cc:288] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 21:33:26.359735 15365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 21:33:26.360976 15365 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [term 3 LEADER]: Becoming Leader. State: Replica: ea13ee710ca24d0ea0388aed191c8022, State: Running, Role: LEADER
I20260926 21:33:26.361532 15365 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } }
I20260926 21:33:26.361891 15361 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:26.364104 15367 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea13ee710ca24d0ea0388aed191c8022. Latest consensus state: current_term: 3 leader_uuid: "ea13ee710ca24d0ea0388aed191c8022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260926 21:33:26.364473 15367 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:26.365116 15366 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ea13ee710ca24d0ea0388aed191c8022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea13ee710ca24d0ea0388aed191c8022" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37687 } } }
I20260926 21:33:26.365408 15366 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea13ee710ca24d0ea0388aed191c8022 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:26.375046 15370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:33:26.384337 15370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:33:26.385979 15370 catalog_manager.cc:1259] Loaded cluster ID: c1939f2808f5474aa556805933fcd8fe
I20260926 21:33:26.386127 15370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 21:33:26.392594 15370 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8324c549ad google::LogMessage::Fail() at ??:0
@ 0x7f8324c5788a google::LogMessage::SendToLog() at ??:0
@ 0x7f8324c544de google::LogMessage::Flush() at ??:0
@ 0x7f8324c55a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f833a687512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f833a684976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f833a70c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f833a70c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f83278a0345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f83278a391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f83278a3799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f832787eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f832ab586db start_thread at ??:0
@ 0x7f8323aee61f clone at ??:0
W20260926 21:33:26.725486 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:26.725798 15363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:26.725910 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:26.733817 15363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:26.733959 15363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:26.734077 15363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:33:26.744491 15363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45959
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:26.745807 15363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:26.747116 15363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:26.760195 15403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:26.760279 15405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:26.760236 15402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:26.761332 15363 server_base.cc:1034] running on GCE node
I20260926 21:33:26.762020 15363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:26.763119 15363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:26.764328 15363 hybrid_clock.cc:648] HybridClock initialized: now 1790458406764252 us; error 62 us; skew 500 ppm
I20260926 21:33:26.764667 15363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:26 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:26.767863 15363 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:33:26.769359 15363 webserver.cc:460] Webserver started at http://127.0.97.1:45959/ using document root <none> and password file <none>
I20260926 21:33:26.770112 15363 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:26.770267 15363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:26.775988 15363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 21:33:26.780947 15412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:26.782320 15363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:33:26.782531 15363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "c3e0f581d0ae4000a0d4d29f7fb158f8"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "99c9c0336f1a02313c8d0b3599677584"
server_key_iv: "3c1c5d10c73b2507f6e391f0ae641337"
server_key_version: "encryptionkey@0"
I20260926 21:33:26.783586 15363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:26.807960 15363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:26.812690 15363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:26.813004 15363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:26.814034 15363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:26.815521 15363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:26.815639 15363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:26.815774 15363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:26.815873 15363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:26.854202 15363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35045
I20260926 21:33:26.854250 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35045 every 8 connection(s)
I20260926 21:33:26.855872 15363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:33:26.861515 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:26.859370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:46141)
0926 21:33:26.860267 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:26.860525 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":348,"threads_started":1}
W20260926 21:33:26.862350 15525 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37687 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37687: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
I20260926 21:33:26.863374 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:26.862988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:41125)
0926 21:33:26.863155 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:26.863247 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:33:26.864358 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:26.863961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:49247)
0926 21:33:26.864138 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:26.864222 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 21:33:26.864684 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 15363
W20260926 21:33:26.864972 15525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 21:33:26.865404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35501
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=37889
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:27.217152 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:27.217476 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:27.217582 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:27.225399 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:27.225555 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:27.225679 15528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:33:27.236001 15528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37889
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:27.237514 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:27.238909 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:27.253156 15536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:27.253218 15534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:27.253252 15533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:27.254107 15528 server_base.cc:1034] running on GCE node
I20260926 21:33:27.254913 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:27.255981 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:27.257256 15528 hybrid_clock.cc:648] HybridClock initialized: now 1790458407257182 us; error 66 us; skew 500 ppm
I20260926 21:33:27.257617 15528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:27 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:27.260888 15528 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:33:27.262612 15528 webserver.cc:460] Webserver started at http://127.0.97.2:37889/ using document root <none> and password file <none>
I20260926 21:33:27.263367 15528 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:27.263520 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:27.269436 15528 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 21:33:27.272505 15543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:27.274030 15528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:33:27.274237 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "85db6b506ee041c5840520b540019d6e"
format_stamp: "Formatted at 2026-09-26 21:33:21 on dist-test-slave-87bt"
server_key: "e1f6c1aa702fcf35fb98d123385f9156"
server_key_iv: "4ddb5a5df434c9dac62c7456104167c1"
server_key_version: "encryptionkey@0"
I20260926 21:33:27.275393 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:27.308012 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:27.313073 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:27.313396 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:27.314488 15528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:27.315918 15528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:27.316056 15528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:27.316188 15528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:27.316280 15528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:27.351656 15528 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35501
I20260926 21:33:27.351748 15655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35501 every 8 connection(s)
I20260926 21:33:27.353226 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:33:27.358667 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:27.356433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:50131)
0926 21:33:27.357269 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:27.357490 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":359,"threads_started":1}
W20260926 21:33:27.359653 15656 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37687 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37687: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
I20260926 21:33:27.359949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 15528
I20260926 21:33:27.360654 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:27.360294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:43759)
0926 21:33:27.360457 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:27.360532 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 21:33:27.360682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37731
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37687
--webserver_port=36515
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:33:27.361763 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:27.361405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:44101)
0926 21:33:27.361569 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:27.361648 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 21:33:27.362075 15656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:33:27.692794 15659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:27.693090 15659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:27.693186 15659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:27.700567 15659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:27.700696 15659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:27.700809 15659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:33:27.710815 15659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36515
--tserver_master_addrs=127.0.97.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:27.712176 15659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:27.713442 15659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:27.727023 15664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:27.727162 15667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:27.727169 15665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:27.728001 15659 server_base.cc:1034] running on GCE node
I20260926 21:33:27.728677 15659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:27.729696 15659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:27.730937 15659 hybrid_clock.cc:648] HybridClock initialized: now 1790458407730881 us; error 42 us; skew 500 ppm
I20260926 21:33:27.731266 15659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:27 dist-test-slave-87bt krb5kdc[14207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:27.734784 15659 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:33:27.736460 15659 webserver.cc:460] Webserver started at http://127.0.97.3:36515/ using document root <none> and password file <none>
I20260926 21:33:27.737185 15659 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:27.737320 15659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:27.742996 15659 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 21:33:27.745891 15674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:27.747337 15659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:33:27.747545 15659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "637ba4b09bd244ff824d0ba01fdb069b"
format_stamp: "Formatted at 2026-09-26 21:33:22 on dist-test-slave-87bt"
server_key: "62b49ea9f0b9364d453b6f16c9210f9c"
server_key_iv: "bfe70370507a1af202981513e717f6e5"
server_key_version: "encryptionkey@0"
I20260926 21:33:27.748569 15659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:27.763342 15659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:27.767719 15659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:27.768033 15659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:27.769088 15659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:27.770561 15659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:27.770676 15659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:27.770808 15659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:27.770895 15659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:27.804577 15659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37731
I20260926 21:33:27.804637 15786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37731 every 8 connection(s)
I20260926 21:33:27.806142 15659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:33:27.811282 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:27.809211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:54189)
0926 21:33:27.810006 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:27.810230 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":291,"threads_started":1}
W20260926 21:33:27.811941 15787 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37687 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37687: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
I20260926 21:33:27.812740 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:27.812433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:49603)
0926 21:33:27.812573 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:27.812639 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 21:33:27.813627 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:27.813339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:35457)
0926 21:33:27.813459 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:27.813529 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:33:27.813829 15787 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 21:33:27.814090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 15659
I20260926 21:33:27.866377 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:27.865580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:42695)
0926 21:33:27.866039 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:27.866131 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":210,"threads_started":1}
I20260926 21:33:28.363554 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:28.362765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:36587)
0926 21:33:28.363216 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:28.363325 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":216,"threads_started":1}
I20260926 21:33:28.815269 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:28.814504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:36807)
0926 21:33:28.814952 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:28.815042 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":206,"threads_started":1}
I20260926 21:33:28.868119 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:28.867394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:46243)
0926 21:33:28.867845 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:28.867948 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":214,"threads_started":1}
I20260926 21:33:29.365442 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:29.364634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:60397)
0926 21:33:29.365154 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:29.365260 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":253,"threads_started":1}
I20260926 21:33:29.817135 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:29.816387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:38397)
0926 21:33:29.816885 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:29.816973 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":280,"threads_started":1}
I20260926 21:33:29.869952 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:29.869241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:34813)
0926 21:33:29.869694 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:29.869777 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":242,"threads_started":1}
I20260926 21:33:30.367513 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:30.366680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:55589)
0926 21:33:30.367218 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:30.367314 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":289,"threads_started":1}
I20260926 21:33:30.818912 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:30.818103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:34901)
0926 21:33:30.818663 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:30.818738 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":345,"threads_started":1}
I20260926 21:33:30.871702 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:30.870950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:39917)
0926 21:33:30.871434 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:30.871520 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260926 21:33:31.369488 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:31.368616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:44447)
0926 21:33:31.369173 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:31.369290 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":229,"threads_started":1}
I20260926 21:33:31.820716 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:31.819955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:40711)
0926 21:33:31.820442 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:31.820526 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":246,"threads_started":1}
I20260926 21:33:31.873624 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:31.872753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:56993)
0926 21:33:31.873320 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:31.873406 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":322,"threads_started":1}
I20260926 21:33:32.371310 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:32.370547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:59689)
0926 21:33:32.371057 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:32.371139 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":292,"threads_started":1}
I20260926 21:33:32.822608 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:32.821867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:47495)
0926 21:33:32.822328 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:32.822411 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260926 21:33:32.875381 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:32.874644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:37733)
0926 21:33:32.875102 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:32.875219 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260926 21:33:33.372830 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:33.372173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:56959)
0926 21:33:33.372590 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:33.372674 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260926 21:33:33.824178 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:33.823464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:56939)
0926 21:33:33.823922 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:33.824016 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260926 21:33:33.877002 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:33.876268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:60549)
0926 21:33:33.876718 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:33.876799 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":198,"threads_started":1}
I20260926 21:33:34.374861 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:34.373923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:34835)
0926 21:33:34.374544 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:34.374642 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
I20260926 21:33:34.826035 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:34.825172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:34635)
0926 21:33:34.825743 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:34.825821 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":279,"threads_started":1}
I20260926 21:33:34.878705 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:34.877925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:51527)
0926 21:33:34.878372 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:34.878528 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260926 21:33:35.376550 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:35.375808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:57369)
0926 21:33:35.376289 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:35.376371 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":221,"threads_started":1}
I20260926 21:33:35.827935 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:35.827143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:51039)
0926 21:33:35.827683 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:35.827772 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260926 21:33:35.880425 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:35.879654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:43103)
0926 21:33:35.880167 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:35.880249 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":279,"threads_started":1}
I20260926 21:33:36.378352 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:36.377463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:49673)
0926 21:33:36.378063 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:36.378178 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":282,"threads_started":1}
I20260926 21:33:36.829689 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:36.828916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:56893)
0926 21:33:36.829421 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:36.829499 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":254,"threads_started":1}
I20260926 21:33:36.882150 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:36.881427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:60619)
0926 21:33:36.881885 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:36.881973 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":248,"threads_started":1}
I20260926 21:33:37.380122 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:37.379339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:36179)
0926 21:33:37.379864 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:37.379945 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":243,"threads_started":1}
I20260926 21:33:37.831627 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:37.830782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:47567)
0926 21:33:37.831321 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:37.831421 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":285,"threads_started":1}
I20260926 21:33:37.883920 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:37.883180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:57773)
0926 21:33:37.883632 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:37.883743 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260926 21:33:38.381819 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:38.381104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:49737)
0926 21:33:38.381542 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:38.381638 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260926 21:33:38.833566 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:38.832604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:58877)
0926 21:33:38.833166 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:38.833298 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":272,"threads_started":1}
I20260926 21:33:38.885640 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:38.884916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:49671)
0926 21:33:38.885376 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:38.885476 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":222,"threads_started":1}
I20260926 21:33:39.383563 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:39.382883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:44859)
0926 21:33:39.383323 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:39.383407 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260926 21:33:39.835543 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:39.834859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:46751)
0926 21:33:39.835310 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:39.835391 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":230,"threads_started":1}
I20260926 21:33:39.887646 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:39.886811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:34937)
0926 21:33:39.887347 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:39.887457 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260926 21:33:40.385211 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:40.384504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:35733)
0926 21:33:40.384946 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:40.385029 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260926 21:33:40.837352 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:40.836596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:33901)
0926 21:33:40.837081 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:40.837167 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":233,"threads_started":1}
I20260926 21:33:40.889290 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:40.888556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:52727)
0926 21:33:40.889004 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:40.889098 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260926 21:33:41.386893 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:41.386067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:40467)
0926 21:33:41.386601 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:41.386708 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":230,"threads_started":1}
I20260926 21:33:41.839003 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:41.838260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:53909)
0926 21:33:41.838746 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:41.838829 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":284,"threads_started":1}
I20260926 21:33:41.891501 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:41.890620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:41871)
0926 21:33:41.891195 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:41.891297 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":312,"threads_started":1}
I20260926 21:33:42.388712 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:42.387982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.2:60489)
0926 21:33:42.388438 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:42.388528 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
I20260926 21:33:42.821192 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 15363
I20260926 21:33:42.841050 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:42.840302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.3:49725)
0926 21:33:42.840783 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:42.840872 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260926 21:33:42.847309 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:42.823408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.0.97.1:35045)
0926 21:33:42.823925 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:42.823940 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:42.823979 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:33:42.825530 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:42.825542 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:42.826034 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:42.826368 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:42.827858 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:42.829201 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:42.830848 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:42.832176 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:42.832462 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:42.840673 (+ 8211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:42.840686 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:42.840710 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:42.840802 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:42.844061 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:42.844849 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:42.844856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:42.844865 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:42.845046 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:42.845405 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:42.845408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:42.845410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:42.845975 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:42.846215 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:42.846645 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 21:33:42.847024 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":235,"threads_started":1}
I20260926 21:33:42.848800 15520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:42.939913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 15363
I20260926 21:33:42.940024 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:42.933963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37687 (local address 127.0.97.1:50773)
0926 21:33:42.939517 (+ 5554us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:42.939679 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5442,"thread_start_us":5319,"threads_started":1}
I20260926 21:33:42.956864 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 15528
I20260926 21:33:42.974642 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:42.957868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.0.97.2:35501)
0926 21:33:42.958540 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:42.958554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:42.958591 (+ 37us) server_negotiation.cc:407] Connection header received
0926 21:33:42.960277 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:42.960287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:42.960786 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:42.961099 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:42.962718 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:42.964022 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:42.965186 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:42.965894 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:42.966045 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:42.968535 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:42.968543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:42.968560 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:42.968598 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:42.971632 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:42.972238 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:42.972245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:42.972251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:42.972442 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:42.972743 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:42.972746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:42.972748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:42.973260 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:42.973510 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:42.973826 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 21:33:42.974297 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":347,"threads_started":1}
I20260926 21:33:42.976011 15651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:43.060946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 15528
I20260926 21:33:43.078715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 15659
I20260926 21:33:43.096707 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:43.079828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.0.97.3:37731)
0926 21:33:43.080478 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:43.080493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:43.080535 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:33:43.082155 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:43.082166 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:43.082692 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:43.082996 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:43.084272 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:43.085572 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:43.087022 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:43.087734 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:43.087923 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:43.090546 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:43.090556 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:43.090575 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:43.090619 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:43.093503 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:43.094213 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:43.094220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:43.094227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:43.094401 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:43.094855 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:43.094859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:43.094861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:43.095396 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:43.095617 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:43.095985 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 21:33:43.096420 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":305,"threads_started":1}
I20260926 21:33:43.098094 15782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:43.181696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 15659
2026-09-26T21:33:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24509 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:33:43 dist-test-slave-87bt krb5kdc[15845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:33:43 dist-test-slave-87bt krb5kdc[15845](info): set up 2 sockets
Sep 26 21:33:43 dist-test-slave-87bt krb5kdc[15845](info): commencing operation
krb5kdc: starting...
W20260926 21:33:44.291003 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:33:44 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:33:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:33:44.539822 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33813
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:44.892575 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:44.892889 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:44.893074 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:44.900540 15858 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 21:33:44.900660 15858 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 21:33:44.900739 15858 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 21:33:44.900822 15858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:33:44.900914 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:44.901048 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:44.901118 15858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:33:44.901201 15858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:33:44.911454 15858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:44.912906 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:44.914320 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:44.927039 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:44.927102 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:44.927011 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:44.928398 15858 server_base.cc:1034] running on GCE node
I20260926 21:33:44.929291 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:44.930905 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:44.932152 15858 hybrid_clock.cc:648] HybridClock initialized: now 1790458424932086 us; error 37 us; skew 500 ppm
I20260926 21:33:44.932561 15858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:44 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:44.935999 15858 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:33:44.937636 15858 webserver.cc:460] Webserver started at http://127.0.97.62:45361/ using document root <none> and password file <none>
I20260926 21:33:44.938381 15858 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:44.938549 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:44.938850 15858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:44.941998 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418"
format_stamp: "Formatted at 2026-09-26 21:33:44 on dist-test-slave-87bt"
server_key: "2ec9910a262600d7fb48af790c1f9316"
server_key_iv: "dc61022b3fb3e537bc4c67369b652aaa"
server_key_version: "encryptionkey@0"
I20260926 21:33:44.942909 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418"
format_stamp: "Formatted at 2026-09-26 21:33:44 on dist-test-slave-87bt"
server_key: "2ec9910a262600d7fb48af790c1f9316"
server_key_iv: "dc61022b3fb3e537bc4c67369b652aaa"
server_key_version: "encryptionkey@0"
I20260926 21:33:44.948817 15858 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 21:33:44.952965 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:44.954800 15858 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 21:33:44.955022 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418"
format_stamp: "Formatted at 2026-09-26 21:33:44 on dist-test-slave-87bt"
server_key: "2ec9910a262600d7fb48af790c1f9316"
server_key_iv: "dc61022b3fb3e537bc4c67369b652aaa"
server_key_version: "encryptionkey@0"
I20260926 21:33:44.955292 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:44.978964 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:44.984313 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:44.984663 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:45.006253 15858 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33813
I20260926 21:33:45.006225 15924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33813 every 8 connection(s)
I20260926 21:33:45.007951 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:33:45.012094 15925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:45.015368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 15858
I20260926 21:33:45.015615 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:33:45.016418 388 external_mini_cluster.cc:1428] Setting key 04e3bb200c0c2afdd16285532635b93c
I20260926 21:33:45.025175 15925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Bootstrap starting.
I20260926 21:33:45.028676 15925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:45.029844 15925 log.cc:824] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:33:45 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:45.033370 15925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: No bootstrap required, opened a new log
I20260926 21:33:45.042335 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:45.018823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.0.97.62:33813)
0926 21:33:45.019641 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:45.019655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:45.019699 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:33:45.021457 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:45.021486 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:45.021959 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:45.022434 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:45.024107 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:45.025531 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:45.027078 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:45.027831 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:45.028054 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:45.032375 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:45.032390 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:45.032415 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:45.032473 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:45.037115 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:45.038003 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:45.038013 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:45.038024 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:45.038265 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:45.038745 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:45.038751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:45.038755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:45.039560 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:45.039900 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:45.040247 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 21:33:45.040776 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":357,"threads_started":1}
I20260926 21:33:45.044727 15925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:33:45.045161 15925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:45.045293 15925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4e2cf1e9a04597b80c44e5a4fd6418, State: Initialized, Role: FOLLOWER
I20260926 21:33:45.045962 15925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:33:45.046235 15925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:33:45.046392 15925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:33:45.046625 15925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:45.049130 15925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:33:45.049755 15925 leader_election.cc:302] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a4e2cf1e9a04597b80c44e5a4fd6418; no voters:
I20260926 21:33:45.050555 15925 leader_election.cc:288] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:33:45.050683 15930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:45.051389 15930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 1 LEADER]: Becoming Leader. State: Replica: 3a4e2cf1e9a04597b80c44e5a4fd6418, State: Running, Role: LEADER
I20260926 21:33:45.052053 15930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:33:45.053018 15925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:33:45.055039 15932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4e2cf1e9a04597b80c44e5a4fd6418. Latest consensus state: current_term: 1 leader_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } } }
I20260926 21:33:45.055469 15932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:45.055974 15931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } } }
I20260926 21:33:45.056285 15931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: This master's current role is: LEADER
I20260926 21:33:45.057297 15937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:33:45.064625 15937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:33:45.075973 15937 catalog_manager.cc:1347] Generated new cluster ID: 92aec389c7734cf8a53fd3949cde9d32
I20260926 21:33:45.076128 15937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:33:45.101682 15937 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:33:45.103343 15937 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:33:45.109894 15937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Generated new TSK 0
I20260926 21:33:45.111021 15937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:33:45.166604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33813
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:45.527902 15951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:45.528193 15951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:45.528357 15951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:45.536170 15951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:45.536306 15951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:45.536523 15951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:33:45.546902 15951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:45.548308 15951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:45.549813 15951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:45.565220 15959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:45.565132 15956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:45.565253 15957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:45.566107 15951 server_base.cc:1034] running on GCE node
I20260926 21:33:45.566915 15951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:45.568063 15951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:45.569325 15951 hybrid_clock.cc:648] HybridClock initialized: now 1790458425569269 us; error 57 us; skew 500 ppm
I20260926 21:33:45.569691 15951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:45 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:45.573092 15951 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:33:45.574852 15951 webserver.cc:460] Webserver started at http://127.0.97.1:37733/ using document root <none> and password file <none>
I20260926 21:33:45.575660 15951 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:45.575821 15951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:45.576169 15951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:45.579600 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "f65e1183b0b14cb1bf5d634442b938b7"
format_stamp: "Formatted at 2026-09-26 21:33:45 on dist-test-slave-87bt"
server_key: "ee644f31753f624606e25e37296d635b"
server_key_iv: "4bbe23df0397f7b503c5e7c448b65105"
server_key_version: "encryptionkey@0"
I20260926 21:33:45.580493 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f65e1183b0b14cb1bf5d634442b938b7"
format_stamp: "Formatted at 2026-09-26 21:33:45 on dist-test-slave-87bt"
server_key: "ee644f31753f624606e25e37296d635b"
server_key_iv: "4bbe23df0397f7b503c5e7c448b65105"
server_key_version: "encryptionkey@0"
I20260926 21:33:45.585880 15951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:33:45.589520 15966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:45.590934 15951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:33:45.591154 15951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "f65e1183b0b14cb1bf5d634442b938b7"
format_stamp: "Formatted at 2026-09-26 21:33:45 on dist-test-slave-87bt"
server_key: "ee644f31753f624606e25e37296d635b"
server_key_iv: "4bbe23df0397f7b503c5e7c448b65105"
server_key_version: "encryptionkey@0"
I20260926 21:33:45.591372 15951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:45.611129 15951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:45.615535 15951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:45.615870 15951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:45.616871 15951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:45.618620 15951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:45.618733 15951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:45.618875 15951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:45.618947 15951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:45.654561 15951 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41671
I20260926 21:33:45.654624 16078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41671 every 8 connection(s)
I20260926 21:33:45.655926 15951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:33:45.662353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 15951
I20260926 21:33:45.662598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:33:45.663617 388 external_mini_cluster.cc:1428] Setting key c44e651b5f15486c2cc8741d03474971
Sep 26 21:33:45 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:45.679378 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:45.658496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59815 (local address 127.0.97.62:33813)
0926 21:33:45.659008 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:45.659012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:45.659987 (+ 975us) server_negotiation.cc:407] Connection header received
0926 21:33:45.661995 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:45.662000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:45.662068 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:45.662164 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:45.664613 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:45.665437 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:45.668082 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:45.668556 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:45.668694 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:45.673329 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:45.673335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:45.673339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:45.673378 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:45.676238 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:45.677003 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:45.677016 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:45.677019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:45.677135 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:45.677575 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:45.677580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:45.677583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:45.677853 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:45.678009 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:45.678767 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 21:33:45.679024 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260926 21:33:45.680394 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:45.659057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:59815)
0926 21:33:45.659807 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:45.659857 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:45.661691 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:45.662485 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:45.662498 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:45.663443 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:45.664369 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:45.664403 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:45.665665 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:45.665669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:45.667889 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:45.667908 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:45.668837 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:45.668847 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:45.669195 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:45.670027 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:45.670068 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:45.673017 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:45.676404 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:45.676415 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:45.676432 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:45.676868 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:45.677246 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:45.677252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:45.677256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:45.677452 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:45.677982 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:45.677990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:45.678336 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 21:33:45.678673 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 21:33:45.679215 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":312,"threads_started":1}
I20260926 21:33:45.682037 16079 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33813
I20260926 21:33:45.682377 16079 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:45.683312 16079 heartbeater.cc:511] Master 127.0.97.62:33813 requested a full tablet report, sending...
I20260926 21:33:45.685930 15890 ts_manager.cc:194] Registered new tserver with Master: f65e1183b0b14cb1bf5d634442b938b7 (127.0.97.1:41671)
I20260926 21:33:45.688351 15890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59815
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:33:45.718341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33813
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:46.052690 16085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:46.052981 16085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:46.053145 16085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:46.060931 16085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:46.061055 16085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:46.061223 16085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:33:46.071590 16085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:46.072973 16085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:46.074244 16085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:46.087442 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:46.087391 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:46.087414 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:46.088254 16085 server_base.cc:1034] running on GCE node
I20260926 21:33:46.088884 16085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:46.089845 16085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:46.091091 16085 hybrid_clock.cc:648] HybridClock initialized: now 1790458426091022 us; error 43 us; skew 500 ppm
I20260926 21:33:46.091444 16085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:46 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:46.094550 16085 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:33:46.096141 16085 webserver.cc:460] Webserver started at http://127.0.97.2:33867/ using document root <none> and password file <none>
I20260926 21:33:46.096836 16085 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:46.096979 16085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:46.097288 16085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:46.100281 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "0b3ba16deddf47708f84d5f056886076"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "0082740178576175b394ce1207d8ad1f"
server_key_iv: "c6f46c8d54c453b6086ea4fd81beff35"
server_key_version: "encryptionkey@0"
I20260926 21:33:46.101038 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0b3ba16deddf47708f84d5f056886076"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "0082740178576175b394ce1207d8ad1f"
server_key_iv: "c6f46c8d54c453b6086ea4fd81beff35"
server_key_version: "encryptionkey@0"
I20260926 21:33:46.106046 16085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:33:46.109423 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:46.110836 16085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:33:46.111001 16085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "0b3ba16deddf47708f84d5f056886076"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "0082740178576175b394ce1207d8ad1f"
server_key_iv: "c6f46c8d54c453b6086ea4fd81beff35"
server_key_version: "encryptionkey@0"
I20260926 21:33:46.111200 16085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:46.123358 16085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:46.127264 16085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:46.127571 16085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:46.128531 16085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:46.129914 16085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:46.130020 16085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:46.130169 16085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:46.130244 16085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:46.163638 16085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40103
I20260926 21:33:46.163688 16212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40103 every 8 connection(s)
I20260926 21:33:46.165009 16085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:33:46.173684 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 16085
I20260926 21:33:46.173895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:33:46.174361 388 external_mini_cluster.cc:1428] Setting key 2aa85e2b527d4b5f99bee4382df28735
Sep 26 21:33:46 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:46.185664 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.167678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41657 (local address 127.0.97.62:33813)
0926 21:33:46.167858 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:46.167862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:46.169189 (+ 1327us) server_negotiation.cc:407] Connection header received
0926 21:33:46.171151 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:46.171156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:46.171254 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:46.171376 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:46.172909 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.173605 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.175360 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.175815 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.175950 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.179654 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:46.179659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:46.179663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:46.179695 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:46.182597 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:46.183334 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:46.183342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:46.183350 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:46.183474 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:46.183914 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:46.183919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:46.183928 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:46.184197 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:46.184355 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:46.185240 (+ 885us) server_negotiation.cc:299] Negotiation successful
0926 21:33:46.185429 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:33:46.187238 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.168219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:41657)
0926 21:33:46.169016 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:46.169056 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:46.170861 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:46.171614 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:46.171622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:46.172158 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:46.172732 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.172749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.173768 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.173772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.175193 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.175209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.176052 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.176059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.176302 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.176985 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:46.177007 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:46.179413 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:46.182743 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:46.182752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:46.182770 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:46.183202 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:46.183590 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:46.183595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:46.183599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:46.183788 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:46.184328 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:46.184343 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:46.184799 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 21:33:46.185109 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 21:33:46.185709 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":362,"threads_started":1}
I20260926 21:33:46.188731 16213 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33813
I20260926 21:33:46.189088 16213 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:46.189838 16213 heartbeater.cc:511] Master 127.0.97.62:33813 requested a full tablet report, sending...
I20260926 21:33:46.191396 15890 ts_manager.cc:194] Registered new tserver with Master: 0b3ba16deddf47708f84d5f056886076 (127.0.97.2:40103)
I20260926 21:33:46.192191 15890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41657
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:33:46.229146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33813
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:33:46.557219 16218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:33:46.557595 16218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:33:46.557788 16218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:33:46.566569 16218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:33:46.566715 16218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:33:46.566848 16218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:33:46.578002 16218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:33:46.579433 16218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:33:46.580823 16218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:33:46.594995 16223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:46.594995 16224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:33:46.595031 16226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:33:46.595526 16218 server_base.cc:1034] running on GCE node
I20260926 21:33:46.596377 16218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:33:46.597379 16218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:33:46.598567 16218 hybrid_clock.cc:648] HybridClock initialized: now 1790458426598510 us; error 44 us; skew 500 ppm
I20260926 21:33:46.598887 16218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:33:46 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:33:46.602191 16218 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:33:46.603753 16218 webserver.cc:460] Webserver started at http://127.0.97.3:44189/ using document root <none> and password file <none>
I20260926 21:33:46.604446 16218 fs_manager.cc:359] Metadata directory not provided
I20260926 21:33:46.604591 16218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:33:46.604902 16218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:33:46.607918 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "299b2665d202496f907e8b05ab3658c9"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "54c4d3711d2a647e6365040d27f400dd"
server_key_iv: "1ee6671770a12de2e9174d6e3c50b43e"
server_key_version: "encryptionkey@0"
I20260926 21:33:46.608737 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "299b2665d202496f907e8b05ab3658c9"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "54c4d3711d2a647e6365040d27f400dd"
server_key_iv: "1ee6671770a12de2e9174d6e3c50b43e"
server_key_version: "encryptionkey@0"
I20260926 21:33:46.614044 16218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:33:46.617571 16233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:46.618942 16218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:33:46.619117 16218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "299b2665d202496f907e8b05ab3658c9"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "54c4d3711d2a647e6365040d27f400dd"
server_key_iv: "1ee6671770a12de2e9174d6e3c50b43e"
server_key_version: "encryptionkey@0"
I20260926 21:33:46.619344 16218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:33:46.638005 16218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:33:46.642397 16218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:33:46.642771 16218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:33:46.643755 16218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:33:46.645170 16218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:33:46.645284 16218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:46.645452 16218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:33:46.645524 16218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:33:46.680106 16218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37583
I20260926 21:33:46.680161 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37583 every 8 connection(s)
I20260926 21:33:46.681406 16218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:33:46.682078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 16218
I20260926 21:33:46.682314 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:33:46.682875 388 external_mini_cluster.cc:1428] Setting key 7eeef95b37004e54494f2e270dde2af7
I20260926 21:33:46.691887 16079 heartbeater.cc:503] Master 127.0.97.62:33813 was elected leader, sending a full tablet report...
Sep 26 21:33:46 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:33:46.701753 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.684195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37579 (local address 127.0.97.62:33813)
0926 21:33:46.684396 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:46.684403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:46.685798 (+ 1395us) server_negotiation.cc:407] Connection header received
0926 21:33:46.687682 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:46.687686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:46.687748 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:46.687827 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:46.689285 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.690024 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.691602 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.692260 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.692430 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.695956 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:46.695960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:46.695963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:46.695993 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:46.699106 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:46.699802 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:46.699806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:46.699808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:46.699927 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:46.700281 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:46.700284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:46.700286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:46.700554 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:46.700661 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:46.701359 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 21:33:46.701524 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:33:46.702819 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.684804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:37579)
0926 21:33:46.685627 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:46.685682 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:46.687444 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:46.688070 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:46.688079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:46.688591 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:33:46.689142 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.689159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.690153 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.690157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.691471 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.691482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.692522 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.692530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.692775 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.693503 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:33:46.693525 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:33:46.695785 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:33:46.699243 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:46.699250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:46.699264 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:46.699696 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:46.700012 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:33:46.700015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:33:46.700017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:33:46.700186 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:33:46.700655 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:33:46.700661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:33:46.701019 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 21:33:46.701242 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 21:33:46.701614 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":300,"threads_started":1}
I20260926 21:33:46.704257 16346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33813
I20260926 21:33:46.704550 16346 heartbeater.cc:464] Registering TS with master...
I20260926 21:33:46.705253 16346 heartbeater.cc:511] Master 127.0.97.62:33813 requested a full tablet report, sending...
I20260926 21:33:46.706681 15890 ts_manager.cc:194] Registered new tserver with Master: 299b2665d202496f907e8b05ab3658c9 (127.0.97.3:37583)
I20260926 21:33:46.707484 15890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37579
I20260926 21:33:46.710573 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:33:46.728492 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.715105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.97.62:33813)
0926 21:33:46.715318 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:46.715326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:46.715698 (+ 372us) server_negotiation.cc:407] Connection header received
0926 21:33:46.715895 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:46.715931 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:46.716024 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:46.716113 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:46.717549 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.718283 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.719622 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.720023 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.720119 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.722670 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:46.722675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:46.722679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:46.722711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:46.725734 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:46.726580 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:46.726584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:46.726587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:46.726707 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:46.727177 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:46.727180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:46.727182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:46.727486 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:46.727683 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:46.728032 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 21:33:46.728204 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:33:46.732971 15890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:33:46.740139 15890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:33:46.771531 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.753134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:53036)
0926 21:33:46.761931 (+ 8797us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:46.761958 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:46.762138 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:46.763697 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:46.763702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:46.763730 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:46.764115 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.764127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.766026 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.766031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.768170 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.768183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.770493 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.770507 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.770756 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.770827 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 21:33:46.770967 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 21:33:46.771178 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8592,"thread_start_us":5614,"threads_started":1}
I20260926 21:33:46.771713 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.753722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.0.97.1:41671)
0926 21:33:46.762804 (+ 9082us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:46.762812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:46.762840 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:33:46.762928 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:46.762936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:46.763209 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:46.763417 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:46.764317 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.765869 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.768334 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.769836 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.769965 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.770209 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:46.771173 (+ 964us) server_negotiation.cc:299] Negotiation successful
0926 21:33:46.771377 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8899,"thread_start_us":8718,"threads_started":1}
I20260926 21:33:46.773962 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.753199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.0.97.3:37583)
0926 21:33:46.761017 (+ 7818us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:46.761032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:46.761082 (+ 50us) server_negotiation.cc:407] Connection header received
0926 21:33:46.761229 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:46.761242 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:46.761623 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:46.761872 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:46.763591 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.765316 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.768272 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.769836 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.769965 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.770210 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:46.773336 (+ 3126us) server_negotiation.cc:299] Negotiation successful
0926 21:33:46.773541 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7561,"thread_start_us":7355,"threads_started":1}
I20260926 21:33:46.773144 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.753136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37583 (local address 127.0.0.1:60976)
0926 21:33:46.759653 (+ 6517us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:46.759711 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:46.759933 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:46.762865 (+ 2932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:46.762874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:46.762907 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:46.763366 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.763382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.765475 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.765481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.768057 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.768075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.772234 (+ 4159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.772244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.772427 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.772454 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 21:33:46.772590 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 21:33:46.772742 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5779,"thread_start_us":8436,"threads_started":1}
I20260926 21:33:46.776607 16014 tablet_service.cc:1467] Processing CreateTablet for tablet d0e27024d6af4273b8e0b40c72f093cf (DEFAULT_TABLE table=test-table [id=6c88327f37f141c6b5febd4ec6c8bb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:46.778239 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.764169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40103 (local address 127.0.0.1:45006)
0926 21:33:46.770848 (+ 6679us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:46.770869 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:46.771003 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:46.772059 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:46.772064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:46.772096 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:46.772570 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.772583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.774231 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.774235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.775976 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:46.775988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.777501 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.777508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:46.777645 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.777669 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 21:33:46.777777 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 21:33:46.777907 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6527,"thread_start_us":6370,"threads_started":1}
I20260926 21:33:46.778314 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:46.762346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.0.97.2:40103)
0926 21:33:46.770373 (+ 8027us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:46.770387 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:46.771006 (+ 619us) server_negotiation.cc:407] Connection header received
0926 21:33:46.771224 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:46.771249 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:46.771696 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:46.771941 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:46.772762 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.774064 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.776131 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:46.777163 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:46.777297 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:46.777505 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:46.777829 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 21:33:46.777999 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7831,"thread_start_us":7667,"threads_started":1}
I20260926 21:33:46.778281 16281 tablet_service.cc:1467] Processing CreateTablet for tablet d0e27024d6af4273b8e0b40c72f093cf (DEFAULT_TABLE table=test-table [id=6c88327f37f141c6b5febd4ec6c8bb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:46.780014 16281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e27024d6af4273b8e0b40c72f093cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:46.780045 16014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e27024d6af4273b8e0b40c72f093cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:46.782075 16148 tablet_service.cc:1467] Processing CreateTablet for tablet d0e27024d6af4273b8e0b40c72f093cf (DEFAULT_TABLE table=test-table [id=6c88327f37f141c6b5febd4ec6c8bb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:33:46.783820 16148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e27024d6af4273b8e0b40c72f093cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:33:46.792353 16364 tablet_bootstrap.cc:492] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Bootstrap starting.
I20260926 21:33:46.795536 16365 tablet_bootstrap.cc:492] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Bootstrap starting.
I20260926 21:33:46.796451 16364 tablet_bootstrap.cc:654] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:46.797842 16364 log.cc:824] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:46.799254 16365 tablet_bootstrap.cc:654] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:46.800328 16367 tablet_bootstrap.cc:492] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Bootstrap starting.
I20260926 21:33:46.800563 16365 log.cc:824] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:46.801394 16364 tablet_bootstrap.cc:492] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: No bootstrap required, opened a new log
I20260926 21:33:46.801828 16364 ts_tablet_manager.cc:1397] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 21:33:46.804431 16367 tablet_bootstrap.cc:654] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Neither blocks nor log segments found. Creating new log.
I20260926 21:33:46.805521 16367 log.cc:824] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Log is configured to *not* fsync() on all Append() calls
I20260926 21:33:46.806033 16365 tablet_bootstrap.cc:492] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: No bootstrap required, opened a new log
I20260926 21:33:46.806376 16365 ts_tablet_manager.cc:1397] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260926 21:33:46.808357 16367 tablet_bootstrap.cc:492] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: No bootstrap required, opened a new log
I20260926 21:33:46.808662 16367 ts_tablet_manager.cc:1397] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 21:33:46.812757 16364 raft_consensus.cc:348] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:46.813323 16364 raft_consensus.cc:374] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:46.813436 16364 raft_consensus.cc:729] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e1183b0b14cb1bf5d634442b938b7, State: Initialized, Role: FOLLOWER
I20260926 21:33:46.814102 16364 consensus_queue.cc:260] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:46.816192 16364 ts_tablet_manager.cc:1428] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 21:33:46.817402 16365 raft_consensus.cc:348] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:46.817878 16365 raft_consensus.cc:374] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:46.817998 16365 raft_consensus.cc:729] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299b2665d202496f907e8b05ab3658c9, State: Initialized, Role: FOLLOWER
I20260926 21:33:46.818563 16367 raft_consensus.cc:348] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:46.818714 16365 consensus_queue.cc:260] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:46.818938 16367 raft_consensus.cc:374] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:33:46.819062 16367 raft_consensus.cc:729] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3ba16deddf47708f84d5f056886076, State: Initialized, Role: FOLLOWER
I20260926 21:33:46.819770 16367 consensus_queue.cc:260] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:46.820106 16346 heartbeater.cc:503] Master 127.0.97.62:33813 was elected leader, sending a full tablet report...
I20260926 21:33:46.820624 16365 ts_tablet_manager.cc:1428] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 21:33:46.821949 16213 heartbeater.cc:503] Master 127.0.97.62:33813 was elected leader, sending a full tablet report...
I20260926 21:33:46.822402 16367 ts_tablet_manager.cc:1428] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260926 21:33:46.908568 16080 tablet.cc:2369] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:33:46.916977 16214 tablet.cc:2369] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:33:46.933234 16347 tablet.cc:2369] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:33:47.112082 16371 raft_consensus.cc:482] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:33:47.112442 16371 raft_consensus.cc:504] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:47.113170 16370 raft_consensus.cc:482] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:33:47.113458 16370 raft_consensus.cc:504] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:47.114691 16371 leader_election.cc:288] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e1183b0b14cb1bf5d634442b938b7 (127.0.97.1:41671), 0b3ba16deddf47708f84d5f056886076 (127.0.97.2:40103)
I20260926 21:33:47.115897 16370 leader_election.cc:288] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 299b2665d202496f907e8b05ab3658c9 (127.0.97.3:37583), 0b3ba16deddf47708f84d5f056886076 (127.0.97.2:40103)
I20260926 21:33:47.122258 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.115112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40103 (local address 127.0.97.3:52393)
0926 21:33:47.115321 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:47.115342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:47.115480 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:47.115880 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:47.115886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:47.115917 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:47.116119 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:47.116130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.117587 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.117592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:47.119696 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:47.119708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.121408 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.121414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:47.121672 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.121718 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 21:33:47.121805 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 21:33:47.121950 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 21:33:47.122959 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.115266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52393 (local address 127.0.97.2:40103)
0926 21:33:47.115521 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:47.115527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:47.115553 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:33:47.115632 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:47.115638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:47.115729 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:47.115835 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:47.116315 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.117411 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.119877 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.121095 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.121227 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.121315 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:47.122564 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0926 21:33:47.122705 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 21:33:47.124037 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e27024d6af4273b8e0b40c72f093cf" candidate_uuid: "299b2665d202496f907e8b05ab3658c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3ba16deddf47708f84d5f056886076" is_pre_election: true
I20260926 21:33:47.124475 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.116318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51675 (local address 127.0.97.3:37583)
0926 21:33:47.116492 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:47.116497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:47.117054 (+ 557us) server_negotiation.cc:407] Connection header received
0926 21:33:47.117275 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:47.117279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:47.117337 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:47.117394 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:47.118094 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.119281 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.121767 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.123026 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.123117 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.123207 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:47.124091 (+ 884us) server_negotiation.cc:299] Negotiation successful
0926 21:33:47.124244 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:33:47.124619 16168 raft_consensus.cc:2454] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299b2665d202496f907e8b05ab3658c9 in term 0.
I20260926 21:33:47.125468 16237 leader_election.cc:302] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b3ba16deddf47708f84d5f056886076, 299b2665d202496f907e8b05ab3658c9; no voters:
I20260926 21:33:47.125859 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.116171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37583 (local address 127.0.97.1:51675)
0926 21:33:47.116963 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:47.116981 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:47.117151 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:47.117587 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:47.117592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:47.117619 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:47.117938 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:47.117948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.119445 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.119450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:47.121584 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:47.121601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.123693 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.123699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:47.123894 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.123932 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 21:33:47.125271 (+ 1339us) client_negotiation.cc:240] Negotiation successful
0926 21:33:47.125411 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":474,"threads_started":1}
I20260926 21:33:47.125927 16371 raft_consensus.cc:2781] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:33:47.126070 16371 raft_consensus.cc:482] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:33:47.126173 16371 raft_consensus.cc:3037] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:47.129653 16371 raft_consensus.cc:504] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:47.130950 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e27024d6af4273b8e0b40c72f093cf" candidate_uuid: "299b2665d202496f907e8b05ab3658c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3ba16deddf47708f84d5f056886076"
I20260926 21:33:47.131181 16168 raft_consensus.cc:3037] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:47.134577 16168 raft_consensus.cc:2454] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299b2665d202496f907e8b05ab3658c9 in term 1.
I20260926 21:33:47.134652 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.117780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60793 (local address 127.0.97.2:40103)
0926 21:33:47.123175 (+ 5395us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:47.123180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:47.127464 (+ 4284us) server_negotiation.cc:407] Connection header received
0926 21:33:47.127532 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:47.127537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:47.127622 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:47.127709 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:47.128624 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.129698 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.132100 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.133421 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.133529 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.133635 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:47.134184 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 21:33:47.134321 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5302,"thread_start_us":13196,"threads_started":1}
I20260926 21:33:47.135046 16237 leader_election.cc:302] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b3ba16deddf47708f84d5f056886076, 299b2665d202496f907e8b05ab3658c9; no voters:
I20260926 21:33:47.135696 16371 leader_election.cc:288] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [CANDIDATE]: Term 1 election: Requested vote from peers f65e1183b0b14cb1bf5d634442b938b7 (127.0.97.1:41671), 0b3ba16deddf47708f84d5f056886076 (127.0.97.2:40103)
I20260926 21:33:47.136029 16371 raft_consensus.cc:2781] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:33:47.136094 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.117160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40103 (local address 127.0.97.1:60793)
0926 21:33:47.126870 (+ 9710us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:47.126890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:47.127005 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:47.128252 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:47.128257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:47.128282 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:47.128455 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:47.128466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.129966 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.129975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:47.131928 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:47.131941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.133776 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.133782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:47.134005 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.134040 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:33:47.135032 (+ 992us) client_negotiation.cc:240] Negotiation successful
0926 21:33:47.135744 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9599,"mutex_wait_us":532,"thread_start_us":18113,"threads_started":1}
I20260926 21:33:47.136430 16371 raft_consensus.cc:686] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 1 LEADER]: Becoming Leader. State: Replica: 299b2665d202496f907e8b05ab3658c9, State: Running, Role: LEADER
I20260926 21:33:47.136655 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.115385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34855 (local address 127.0.97.1:41671)
0926 21:33:47.115590 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:47.115595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:47.128254 (+ 12659us) server_negotiation.cc:407] Connection header received
0926 21:33:47.128342 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:47.128348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:47.128447 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:47.128553 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:33:47.129258 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.130680 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.133631 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.135427 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.135559 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.135675 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:47.136291 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 21:33:47.136440 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":67}
I20260926 21:33:47.136737 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e27024d6af4273b8e0b40c72f093cf" candidate_uuid: "f65e1183b0b14cb1bf5d634442b938b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3ba16deddf47708f84d5f056886076" is_pre_election: true
I20260926 21:33:47.137060 16168 raft_consensus.cc:2379] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f65e1183b0b14cb1bf5d634442b938b7 in current term 1: Already voted for candidate 299b2665d202496f907e8b05ab3658c9 in this term.
I20260926 21:33:47.137349 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.115112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.97.3:34855)
0926 21:33:47.127986 (+ 12874us) negotiation.cc:107] Waiting for socket to connect
0926 21:33:47.128003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:33:47.128115 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:33:47.128744 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:33:47.128752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:33:47.128789 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:33:47.129064 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:47.129079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.130889 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.130898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:47.133370 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:33:47.133391 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.135838 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.135855 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:33:47.136090 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.136136 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 21:33:47.136816 (+ 680us) client_negotiation.cc:240] Negotiation successful
0926 21:33:47.136975 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12762,"mutex_wait_us":853,"thread_start_us":14646,"threads_started":1}
I20260926 21:33:47.137748 16371 consensus_queue.cc:237] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } }
I20260926 21:33:47.138405 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e27024d6af4273b8e0b40c72f093cf" candidate_uuid: "299b2665d202496f907e8b05ab3658c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e1183b0b14cb1bf5d634442b938b7"
I20260926 21:33:47.138377 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e27024d6af4273b8e0b40c72f093cf" candidate_uuid: "299b2665d202496f907e8b05ab3658c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e1183b0b14cb1bf5d634442b938b7" is_pre_election: true
I20260926 21:33:47.138821 16033 raft_consensus.cc:3037] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:33:47.139886 16301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e27024d6af4273b8e0b40c72f093cf" candidate_uuid: "f65e1183b0b14cb1bf5d634442b938b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299b2665d202496f907e8b05ab3658c9" is_pre_election: true
I20260926 21:33:47.140920 15970 leader_election.cc:302] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65e1183b0b14cb1bf5d634442b938b7; no voters: 0b3ba16deddf47708f84d5f056886076, 299b2665d202496f907e8b05ab3658c9
I20260926 21:33:47.141994 16033 raft_consensus.cc:2454] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299b2665d202496f907e8b05ab3658c9 in term 1.
I20260926 21:33:47.142321 16370 raft_consensus.cc:2726] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 21:33:47.143633 15890 catalog_manager.cc:5495] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 299b2665d202496f907e8b05ab3658c9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "299b2665d202496f907e8b05ab3658c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } health_report { overall_health: UNKNOWN } } }
I20260926 21:33:47.229110 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.223291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.0.97.3:37583)
0926 21:33:47.223498 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:47.223505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:47.223638 (+ 133us) server_negotiation.cc:407] Connection header received
0926 21:33:47.223927 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:47.223931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:47.224030 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:47.224206 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:33:47.224616 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.225375 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.226891 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.227301 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.227443 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.227980 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:33:47.228491 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:33:47.228670 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:47.228717 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 21:33:47.228803 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:33:47.237383 16033 raft_consensus.cc:1261] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Refusing update from remote peer 299b2665d202496f907e8b05ab3658c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:47.237387 16168 raft_consensus.cc:1261] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 1 FOLLOWER]: Refusing update from remote peer 299b2665d202496f907e8b05ab3658c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:33:47.238261 16377 consensus_queue.cc:1035] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:47.238724 16371 consensus_queue.cc:1035] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:33:47.250818 16381 mvcc.cc:205] Tried to move back new op lower bound from 7333717717954056192 to 7333717717562679296. Current Snapshot: MvccSnapshot[applied={T|T < 7333717717954056192}]
I20260926 21:33:47.251741 16380 mvcc.cc:205] Tried to move back new op lower bound from 7333717717954056192 to 7333717717562679296. Current Snapshot: MvccSnapshot[applied={T|T < 7333717717954056192}]
I20260926 21:33:47.267447 15890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:33:47.267680 15890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:33:47.271301 15890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Sending DeleteTablet for 3 replicas of tablet d0e27024d6af4273b8e0b40c72f093cf
I20260926 21:33:47.272620 16281 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e27024d6af4273b8e0b40c72f093cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60976
I20260926 21:33:47.272606 16148 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e27024d6af4273b8e0b40c72f093cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45006
I20260926 21:33:47.272604 16014 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e27024d6af4273b8e0b40c72f093cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:33:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53036
I20260926 21:33:47.273954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 15951
I20260926 21:33:47.301188 16392 tablet_replica.cc:330] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: stopping tablet replica
I20260926 21:33:47.305192 16392 raft_consensus.cc:2229] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:47.306006 16392 raft_consensus.cc:2258] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:47.308259 16392 ts_tablet_manager.cc:1905] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:47.314647 16391 tablet_replica.cc:330] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: stopping tablet replica
I20260926 21:33:47.315181 16391 raft_consensus.cc:2229] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:33:47.316152 16390 tablet_replica.cc:330] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: stopping tablet replica
I20260926 21:33:47.316375 16391 raft_consensus.cc:2258] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:33:47.316596 16390 raft_consensus.cc:2229] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:33:47.317821 16390 raft_consensus.cc:2258] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 21:33:47 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:33:47.319523 16391 ts_tablet_manager.cc:1905] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:47.321771 16390 ts_tablet_manager.cc:1905] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:33:47.326828 16392 ts_tablet_manager.cc:1918] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:47.327014 16392 log.cc:1196] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/d0e27024d6af4273b8e0b40c72f093cf
I20260926 21:33:47.327204 16391 ts_tablet_manager.cc:1918] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:47.327382 16391 log.cc:1196] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/d0e27024d6af4273b8e0b40c72f093cf
I20260926 21:33:47.327543 16392 ts_tablet_manager.cc:1939] T d0e27024d6af4273b8e0b40c72f093cf P f65e1183b0b14cb1bf5d634442b938b7: Deleting consensus metadata
I20260926 21:33:47.327874 16391 ts_tablet_manager.cc:1939] T d0e27024d6af4273b8e0b40c72f093cf P 0b3ba16deddf47708f84d5f056886076: Deleting consensus metadata
I20260926 21:33:47.329541 15874 catalog_manager.cc:4841] TS f65e1183b0b14cb1bf5d634442b938b7 (127.0.97.1:41671): tablet d0e27024d6af4273b8e0b40c72f093cf (table test-table [id=6c88327f37f141c6b5febd4ec6c8bb31]) successfully deleted
I20260926 21:33:47.329546 15877 catalog_manager.cc:4841] TS 0b3ba16deddf47708f84d5f056886076 (127.0.97.2:40103): tablet d0e27024d6af4273b8e0b40c72f093cf (table test-table [id=6c88327f37f141c6b5febd4ec6c8bb31]) successfully deleted
I20260926 21:33:47.331266 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.275306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.0.97.1:41671)
0926 21:33:47.275494 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:47.275501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:47.298104 (+ 22603us) server_negotiation.cc:407] Connection header received
0926 21:33:47.298578 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:47.298586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:47.298703 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:47.298781 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:47.301454 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.302638 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.308494 (+ 5856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.309152 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.309255 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.319486 (+ 10231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:47.319503 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:47.319526 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:47.319573 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:47.323344 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:47.324401 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:47.324411 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:47.324415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:47.324594 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:47.329410 (+ 4816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:47.329416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:47.329419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:47.329928 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:47.330278 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:47.330652 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 21:33:47.330888 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:33:47.331291 16390 ts_tablet_manager.cc:1918] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:33:47.331857 16390 log.cc:1196] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/d0e27024d6af4273b8e0b40c72f093cf
I20260926 21:33:47.332055 16074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:47.332422 16390 ts_tablet_manager.cc:1939] T d0e27024d6af4273b8e0b40c72f093cf P 299b2665d202496f907e8b05ab3658c9: Deleting consensus metadata
I20260926 21:33:47.334860 15877 catalog_manager.cc:4841] TS 299b2665d202496f907e8b05ab3658c9 (127.0.97.3:37583): tablet d0e27024d6af4273b8e0b40c72f093cf (table test-table [id=6c88327f37f141c6b5febd4ec6c8bb31]) successfully deleted
I20260926 21:33:47.443745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 15951
I20260926 21:33:47.462838 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 16085
Sep 26 21:33:47 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:33:47.478672 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.463681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.0.97.2:40103)
0926 21:33:47.463899 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:47.463904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:47.463926 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:33:47.464052 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:47.464062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:47.464137 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:47.464247 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:47.465521 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.466222 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.467635 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.468049 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.468186 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.472658 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:47.472676 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:47.472691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:47.472733 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:47.475839 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:47.476574 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:47.476583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:47.476585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:47.476716 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:47.477082 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:47.477087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:47.477090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:47.477441 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:47.477743 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:47.478063 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 21:33:47.478320 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:33:47.479331 16208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:47.574901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 16085
I20260926 21:33:47.591931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 16218
Sep 26 21:33:47 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:33:47.609928 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:33:47.592777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.0.97.3:37583)
0926 21:33:47.593023 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 21:33:47.593030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:33:47.593059 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:33:47.593133 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:33:47.593138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:33:47.593233 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:33:47.593362 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:33:47.594658 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.595812 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.597156 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:33:47.597905 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:33:47.598053 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:33:47.602860 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:33:47.602882 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:33:47.602903 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:33:47.602962 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:33:47.606669 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:47.607602 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:47.607611 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:47.607613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:47.607751 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:33:47.608186 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:33:47.608189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:33:47.608191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:33:47.608627 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:33:47.608894 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 21:33:47.609334 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 21:33:47.609561 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 21:33:47.610879 16341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:47.734088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 16218
I20260926 21:33:47.756157 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 15858
I20260926 21:33:47.757241 15920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:33:47.866636 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 15858
I20260926 21:34:07.888556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33813
--webserver_interface=127.0.97.62
--webserver_port=45361
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:08.230682 16398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:08.231081 16398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:08.231259 16398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:08.239068 16398 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 21:34:08.239230 16398 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 21:34:08.239341 16398 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 21:34:08.239449 16398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:34:08.239557 16398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:08.239691 16398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:08.239786 16398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:34:08.239876 16398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:34:08.250581 16398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:08.252085 16398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:08.254105 16398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:08.278208 16404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:08.278241 16405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:08.278520 16398 server_base.cc:1034] running on GCE node
W20260926 21:34:08.278263 16407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:08.279822 16398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:08.281375 16398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:08.282648 16398 hybrid_clock.cc:648] HybridClock initialized: now 1790458448282578 us; error 28 us; skew 500 ppm
I20260926 21:34:08.283136 16398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:08 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:08.287110 16398 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:34:08.289134 16398 webserver.cc:460] Webserver started at http://127.0.97.62:45361/ using document root <none> and password file <none>
I20260926 21:34:08.290006 16398 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:08.290153 16398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:08.296610 16398 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260926 21:34:08.299861 16414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:08.301309 16398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:34:08.301520 16398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418"
format_stamp: "Formatted at 2026-09-26 21:33:44 on dist-test-slave-87bt"
server_key: "2ec9910a262600d7fb48af790c1f9316"
server_key_iv: "dc61022b3fb3e537bc4c67369b652aaa"
server_key_version: "encryptionkey@0"
I20260926 21:34:08.302757 16398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:08.326098 16398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:08.330624 16398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:08.330968 16398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:08.350977 16398 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33813
I20260926 21:34:08.351027 16465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33813 every 8 connection(s)
I20260926 21:34:08.352635 16398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:34:08.353272 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 16398
I20260926 21:34:08.354270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41671
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33813
--webserver_port=37733
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:34:08.360051 16466 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:34:08.363433 16466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Bootstrap starting.
I20260926 21:34:08.383376 16466 log.cc:824] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:08.395529 16466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:34:08.396140 16466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Bootstrap complete.
I20260926 21:34:08.405490 16466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:34:08.406461 16466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4e2cf1e9a04597b80c44e5a4fd6418, State: Initialized, Role: FOLLOWER
I20260926 21:34:08.407294 16466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:34:08.407626 16466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:34:08.407830 16466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:34:08.408039 16466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:34:08.412366 16466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:34:08.413034 16466 leader_election.cc:302] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a4e2cf1e9a04597b80c44e5a4fd6418; no voters:
I20260926 21:34:08.413781 16466 leader_election.cc:288] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 21:34:08.413933 16470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:34:08.416436 16466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:34:08.421765 16470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 2 LEADER]: Becoming Leader. State: Replica: 3a4e2cf1e9a04597b80c44e5a4fd6418, State: Running, Role: LEADER
I20260926 21:34:08.422555 16470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:34:08.425794 16471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } } }
I20260926 21:34:08.426222 16471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:08.426688 16472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4e2cf1e9a04597b80c44e5a4fd6418. Latest consensus state: current_term: 2 leader_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } } }
I20260926 21:34:08.427150 16472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:08.432964 16483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:34:08.442409 16483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:34:08.444684 16483 catalog_manager.cc:1259] Loaded cluster ID: 92aec389c7734cf8a53fd3949cde9d32
I20260926 21:34:08.444964 16483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:34:08.451287 16483 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:34:08.453814 16483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Loaded TSK: 0
I20260926 21:34:08.475767 16483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Generated new TSK 1
I20260926 21:34:08.476168 16483 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 21:34:08.728889 16468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:08.729259 16468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:08.729444 16468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:08.737210 16468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:08.737399 16468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:08.737593 16468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:34:08.747903 16468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37733
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:08.749336 16468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:08.750743 16468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:08.766409 16494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:08.766482 16496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:08.767424 16468 server_base.cc:1034] running on GCE node
W20260926 21:34:08.767678 16493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:08.768571 16468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:08.769852 16468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:08.771119 16468 hybrid_clock.cc:648] HybridClock initialized: now 1790458448771044 us; error 63 us; skew 500 ppm
I20260926 21:34:08.771516 16468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:08 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:08.781427 16468 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:34:08.783413 16468 webserver.cc:460] Webserver started at http://127.0.97.1:37733/ using document root <none> and password file <none>
I20260926 21:34:08.784382 16468 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:08.784554 16468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:08.792042 16468 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 21:34:08.795329 16503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:08.797025 16468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:34:08.797200 16468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "f65e1183b0b14cb1bf5d634442b938b7"
format_stamp: "Formatted at 2026-09-26 21:33:45 on dist-test-slave-87bt"
server_key: "ee644f31753f624606e25e37296d635b"
server_key_iv: "4bbe23df0397f7b503c5e7c448b65105"
server_key_version: "encryptionkey@0"
I20260926 21:34:08.798606 16468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:08.820412 16468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:08.826314 16468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:08.826651 16468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:08.828027 16468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:08.829836 16468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:08.829981 16468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:08.830116 16468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:08.830201 16468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:08.869813 16468 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41671
I20260926 21:34:08.869850 16615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41671 every 8 connection(s)
I20260926 21:34:08.871680 16468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:34:08.879163 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 16468
I20260926 21:34:08.879963 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40103
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33813
--webserver_port=33867
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:34:08 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:08.898211 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:08.874895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:60565)
0926 21:34:08.875663 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:08.875700 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:08.877386 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:08.878884 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:08.878895 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:08.879675 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:08.880475 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:08.880502 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:08.882146 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:08.882154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:08.884285 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:08.884304 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:08.885396 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:08.885406 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:08.885763 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:08.886539 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:08.886575 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:08.889148 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:08.893942 (+ 4794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:08.893949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:08.893975 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:08.894408 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:08.895012 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:08.895016 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:08.895018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:08.895213 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:08.896223 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:08.896229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:08.896549 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 21:34:08.896772 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 21:34:08.897173 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":314,"threads_started":1}
I20260926 21:34:08.898833 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:08.874576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60565 (local address 127.0.97.62:33813)
0926 21:34:08.875224 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:08.875245 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:08.875844 (+ 599us) server_negotiation.cc:407] Connection header received
0926 21:34:08.877903 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:08.877916 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:08.878252 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:08.878637 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:08.880714 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:08.881963 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:08.884449 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:08.885163 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:08.885304 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:08.889497 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:08.889511 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:08.889535 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:08.889591 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:08.893770 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:08.894633 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:08.894646 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:08.894657 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:08.894867 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:08.895353 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:08.895361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:08.895367 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:08.896079 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:08.896279 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:08.897057 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 21:34:08.897532 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":267,"threads_started":1}
I20260926 21:34:08.902307 16616 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33813
I20260926 21:34:08.902757 16616 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:08.903815 16616 heartbeater.cc:511] Master 127.0.97.62:33813 requested a full tablet report, sending...
I20260926 21:34:08.906607 16431 ts_manager.cc:194] Registered new tserver with Master: f65e1183b0b14cb1bf5d634442b938b7 (127.0.97.1:41671)
I20260926 21:34:08.908270 16431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60565
W20260926 21:34:09.221688 16620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:09.222000 16620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:09.222170 16620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:09.230283 16620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:09.230515 16620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:09.230711 16620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:34:09.241039 16620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33867
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:09.242489 16620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:09.243867 16620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:09.256986 16625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:09.257099 16628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:09.257160 16626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:09.258038 16620 server_base.cc:1034] running on GCE node
I20260926 21:34:09.258733 16620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:09.259692 16620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:09.260957 16620 hybrid_clock.cc:648] HybridClock initialized: now 1790458449260887 us; error 51 us; skew 500 ppm
I20260926 21:34:09.261288 16620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:09 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:09.264467 16620 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:34:09.265936 16620 webserver.cc:460] Webserver started at http://127.0.97.2:33867/ using document root <none> and password file <none>
I20260926 21:34:09.266731 16620 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:09.266885 16620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:09.272382 16620 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:34:09.275108 16635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:09.276450 16620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:34:09.276623 16620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "0b3ba16deddf47708f84d5f056886076"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "0082740178576175b394ce1207d8ad1f"
server_key_iv: "c6f46c8d54c453b6086ea4fd81beff35"
server_key_version: "encryptionkey@0"
I20260926 21:34:09.277607 16620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:09.296898 16620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:09.301535 16620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:09.301818 16620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:09.302960 16620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:09.304461 16620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:09.304575 16620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:09.304713 16620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:09.304802 16620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:09.339908 16620 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40103
I20260926 21:34:09.339962 16747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40103 every 8 connection(s)
I20260926 21:34:09.341467 16620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:34:09.348932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 16620
I20260926 21:34:09.349709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37583
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33813
--webserver_port=44189
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:34:09 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:09.367517 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.344497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46211 (local address 127.0.97.62:33813)
0926 21:34:09.344694 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.344698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.345966 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 21:34:09.347996 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.348002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.348103 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.348233 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:09.350597 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.351706 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.354349 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.355083 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.355210 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.359866 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:09.359873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:09.359878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:09.359920 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:09.363523 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:09.364601 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:09.364608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:09.364611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:09.364765 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:09.365322 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:09.365326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:09.365329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:09.365748 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:09.365895 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.366900 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.367172 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:34:09.368984 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.344980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:46211)
0926 21:34:09.345769 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:09.345812 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:09.347692 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:09.348535 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:09.348549 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:09.349365 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:09.350272 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.350313 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.351926 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.351934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.354160 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.354179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.355365 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.355374 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.355706 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.356563 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:09.356605 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:09.359562 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:09.363707 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:09.363718 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:09.363739 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:09.364448 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:09.364896 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:09.364904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:09.364910 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:09.365175 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:09.365907 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:09.365919 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:09.366444 (+ 525us) client_negotiation.cc:769] Sending connection context
0926 21:34:09.366776 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 21:34:09.367392 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":353,"threads_started":1}
I20260926 21:34:09.370611 16748 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33813
I20260926 21:34:09.370980 16748 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:09.372017 16748 heartbeater.cc:511] Master 127.0.97.62:33813 requested a full tablet report, sending...
I20260926 21:34:09.373716 16431 ts_manager.cc:194] Registered new tserver with Master: 0b3ba16deddf47708f84d5f056886076 (127.0.97.2:40103)
I20260926 21:34:09.374650 16431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46211
W20260926 21:34:09.694679 16751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:09.694980 16751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:09.695147 16751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:09.702606 16751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:09.702715 16751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:09.702921 16751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:34:09.713088 16751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44189
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:09.714469 16751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:09.715849 16751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:09.730043 16757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:09.730018 16759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:09.730356 16756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:09.730852 16751 server_base.cc:1034] running on GCE node
I20260926 21:34:09.731505 16751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:09.732515 16751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:09.733759 16751 hybrid_clock.cc:648] HybridClock initialized: now 1790458449733711 us; error 44 us; skew 500 ppm
I20260926 21:34:09.734108 16751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:09 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:09.737634 16751 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:34:09.739343 16751 webserver.cc:460] Webserver started at http://127.0.97.3:44189/ using document root <none> and password file <none>
I20260926 21:34:09.740052 16751 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:09.740171 16751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:09.745558 16751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:34:09.748440 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:09.749925 16751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:34:09.750074 16751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "299b2665d202496f907e8b05ab3658c9"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "54c4d3711d2a647e6365040d27f400dd"
server_key_iv: "1ee6671770a12de2e9174d6e3c50b43e"
server_key_version: "encryptionkey@0"
I20260926 21:34:09.751133 16751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:09.779745 16751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:09.784287 16751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:09.784590 16751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:09.785605 16751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:09.786974 16751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:09.787061 16751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:09.787132 16751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:09.787187 16751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:09.822641 16751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37583
I20260926 21:34:09.822687 16878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37583 every 8 connection(s)
I20260926 21:34:09.824203 16751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:34:09.829149 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 16751
Sep 26 21:34:09 dist-test-slave-87bt krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:09.845706 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.830175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.0.97.62:33813)
0926 21:34:09.831088 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.831093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.831110 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:34:09.831164 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.831168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.831217 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.831338 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:09.833023 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.833860 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.835384 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.835972 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.836124 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.839434 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:09.839440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:09.839443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:09.839481 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:09.842845 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:09.843605 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:09.843616 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:09.843618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:09.843741 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:09.844175 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:09.844185 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:09.844187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:09.844461 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:09.844741 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.845082 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.845307 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":782,"thread_start_us":645,"threads_started":1}
I20260926 21:34:09.847435 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.827641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:49797)
0926 21:34:09.828354 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:09.828404 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:09.830523 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:09.831526 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:09.831535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:09.832086 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:09.832666 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.832684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.833888 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.833893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.835275 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.835287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.836336 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.836343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.836583 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.837301 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:09.837323 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:09.839783 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:09.843148 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:09.843158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:09.843178 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:09.843731 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:09.844118 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:09.844122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:09.844126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:09.844321 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:09.844907 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:09.844916 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:09.845272 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 21:34:09.845589 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 21:34:09.846166 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":296,"threads_started":1}
I20260926 21:34:09.848554 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.827151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49797 (local address 127.0.97.62:33813)
0926 21:34:09.827341 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.827345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.828543 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 21:34:09.830936 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.830943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.831020 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.831079 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:09.832864 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.833684 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.835449 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.835979 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.836127 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.840033 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:09.840039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:09.840042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:09.840082 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:09.842993 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:09.843867 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:09.843873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:09.843881 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:09.843992 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:09.844456 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:09.844461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:09.844468 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:09.844767 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:09.844919 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.848028 (+ 3109us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.848287 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":70}
I20260926 21:34:09.849644 16879 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33813
I20260926 21:34:09.849920 16879 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:09.850697 16879 heartbeater.cc:511] Master 127.0.97.62:33813 requested a full tablet report, sending...
I20260926 21:34:09.852164 16431 ts_manager.cc:194] Registered new tserver with Master: 299b2665d202496f907e8b05ab3658c9 (127.0.97.3:37583)
I20260926 21:34:09.853397 16431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49797
I20260926 21:34:09.860522 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:34:09.866854 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.861340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.0.97.62:33813)
0926 21:34:09.861519 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.861523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.862043 (+ 520us) server_negotiation.cc:407] Connection header received
0926 21:34:09.862218 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.862222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.862355 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.862554 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:34:09.862921 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.863637 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.865149 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.865538 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.865627 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.866073 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:34:09.866278 (+ 205us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 21:34:09.866658 (+ 380us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52528: token expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 21:34:09.866961 16618 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52528: token expired
W20260926 21:34:09.867537 16350 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:33813)
I20260926 21:34:09.881101 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.868027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.0.97.62:33813)
0926 21:34:09.868219 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.868226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.868242 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:34:09.868407 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.868412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.868483 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.868587 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:09.870048 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.870870 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.872289 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.872688 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.872831 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.875842 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:09.875853 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:09.875857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:09.875892 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:09.878380 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:09.879108 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:09.879114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:09.879117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:09.879243 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:09.879637 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:09.879643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:09.879646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:09.879961 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:09.880218 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.880543 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.880760 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:34:09.884693 16431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:34:09.891940 16431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:34:09.912552 16616 heartbeater.cc:503] Master 127.0.97.62:33813 was elected leader, sending a full tablet report...
I20260926 21:34:09.918745 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.904445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39550)
0926 21:34:09.905803 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:09.905853 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:09.906076 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:09.908011 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:09.908020 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:09.908059 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:09.908606 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.908624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.911586 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.911596 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.915506 (+ 3910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.915528 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.917579 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.917588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.917826 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.917896 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 21:34:09.918017 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 21:34:09.918241 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":70,"thread_start_us":993,"threads_started":1}
I20260926 21:34:09.919859 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.904822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40103 (local address 127.0.0.1:42902)
0926 21:34:09.907264 (+ 2442us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:09.907279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:09.907453 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:09.908735 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:09.908743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:09.908779 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:09.909240 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.909256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.911867 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.911876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.916736 (+ 4860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.916747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.919086 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.919092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.919259 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.919293 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:34:09.919396 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 21:34:09.919541 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2334,"thread_start_us":2189,"threads_started":1}
I20260926 21:34:09.919962 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.905272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.0.97.2:40103)
0926 21:34:09.906820 (+ 1548us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.906833 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.907792 (+ 959us) server_negotiation.cc:407] Connection header received
0926 21:34:09.907948 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.907958 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.908209 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.908426 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:09.909472 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.911587 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.916921 (+ 5334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.918561 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.918692 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.918892 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.919468 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.919663 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":1204,"threads_started":1}
I20260926 21:34:09.920672 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.904708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.0.97.1:41671)
0926 21:34:09.907248 (+ 2540us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.907257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.907351 (+ 94us) server_negotiation.cc:407] Connection header received
0926 21:34:09.907473 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.907481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.907749 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.908896 (+ 1147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:09.909208 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.911371 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.915701 (+ 4330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.917271 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.919273 (+ 2002us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.919595 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.919667 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.920283 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2371,"thread_start_us":2225,"threads_started":1}
I20260926 21:34:09.924253 16683 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3ad5f3352f4080aaeb92408affe2bc (DEFAULT_TABLE table=test-table [id=46416b5344d04bf1b39eb7396957e6ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:09.924412 16551 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3ad5f3352f4080aaeb92408affe2bc (DEFAULT_TABLE table=test-table [id=46416b5344d04bf1b39eb7396957e6ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:09.925735 16683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3ad5f3352f4080aaeb92408affe2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:09.925936 16551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3ad5f3352f4080aaeb92408affe2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:09.926596 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.908214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.0.97.3:37583)
0926 21:34:09.915976 (+ 7762us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.915987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.916488 (+ 501us) server_negotiation.cc:407] Connection header received
0926 21:34:09.916639 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.916664 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.916976 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.917189 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:09.917897 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.919745 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.923481 (+ 3736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.924929 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.925086 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.925351 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.925977 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.926235 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7556,"thread_start_us":7047,"threads_started":1}
I20260926 21:34:09.927294 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.907938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37583 (local address 127.0.0.1:39984)
0926 21:34:09.916322 (+ 8384us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:09.916343 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:09.916496 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:09.917206 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:09.917211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:09.917239 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:09.917687 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.917699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.920125 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.920130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.923249 (+ 3119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.923267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.925557 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.925567 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.925749 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.925787 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 21:34:09.925890 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 21:34:09.926047 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8239,"thread_start_us":7529,"threads_started":1}
I20260926 21:34:09.931772 16814 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3ad5f3352f4080aaeb92408affe2bc (DEFAULT_TABLE table=test-table [id=46416b5344d04bf1b39eb7396957e6ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:09.933862 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3ad5f3352f4080aaeb92408affe2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:09.939267 16895 tablet_bootstrap.cc:492] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: Bootstrap starting.
I20260926 21:34:09.941561 16896 tablet_bootstrap.cc:492] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Bootstrap starting.
I20260926 21:34:09.943548 16895 tablet_bootstrap.cc:654] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:09.944968 16895 log.cc:824] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:09.945292 16896 tablet_bootstrap.cc:654] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:09.946520 16896 log.cc:824] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:09.948307 16895 tablet_bootstrap.cc:492] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: No bootstrap required, opened a new log
I20260926 21:34:09.948609 16895 ts_tablet_manager.cc:1397] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 21:34:09.949714 16896 tablet_bootstrap.cc:492] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: No bootstrap required, opened a new log
I20260926 21:34:09.950011 16896 ts_tablet_manager.cc:1397] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 21:34:09.951931 16899 tablet_bootstrap.cc:492] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: Bootstrap starting.
I20260926 21:34:09.955765 16899 tablet_bootstrap.cc:654] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:09.957109 16899 log.cc:824] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:09.958969 16896 raft_consensus.cc:348] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.958984 16895 raft_consensus.cc:348] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.959370 16896 raft_consensus.cc:374] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:09.959438 16895 raft_consensus.cc:374] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:09.959460 16896 raft_consensus.cc:729] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65e1183b0b14cb1bf5d634442b938b7, State: Initialized, Role: FOLLOWER
I20260926 21:34:09.959553 16895 raft_consensus.cc:729] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3ba16deddf47708f84d5f056886076, State: Initialized, Role: FOLLOWER
I20260926 21:34:09.960111 16896 consensus_queue.cc:260] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.960809 16899 tablet_bootstrap.cc:492] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: No bootstrap required, opened a new log
I20260926 21:34:09.961146 16899 ts_tablet_manager.cc:1397] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 21:34:09.963907 16896 ts_tablet_manager.cc:1428] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 21:34:09.964871 16895 consensus_queue.cc:260] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.965471 16901 raft_consensus.cc:482] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:34:09.965772 16901 raft_consensus.cc:504] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.966212 16748 heartbeater.cc:503] Master 127.0.97.62:33813 was elected leader, sending a full tablet report...
I20260926 21:34:09.967782 16901 leader_election.cc:288] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65e1183b0b14cb1bf5d634442b938b7 (127.0.97.1:41671), 299b2665d202496f907e8b05ab3658c9 (127.0.97.3:37583)
I20260926 21:34:09.969559 16895 ts_tablet_manager.cc:1428] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: Time spent starting tablet: real 0.021s user 0.014s sys 0.005s
I20260926 21:34:09.975523 16899 raft_consensus.cc:348] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.976068 16899 raft_consensus.cc:374] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:09.976171 16899 raft_consensus.cc:729] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299b2665d202496f907e8b05ab3658c9, State: Initialized, Role: FOLLOWER
I20260926 21:34:09.976960 16899 consensus_queue.cc:260] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.978001 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.968686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35855 (local address 127.0.97.1:41671)
0926 21:34:09.968890 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.968899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.970371 (+ 1472us) server_negotiation.cc:407] Connection header received
0926 21:34:09.970652 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.970658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.970733 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.970795 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:09.971610 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.972805 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.975458 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.976681 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.976795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.976895 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.977596 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.977746 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 21:34:09.978288 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.968115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.97.2:35855)
0926 21:34:09.970267 (+ 2152us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:09.970287 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:09.970493 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:09.970991 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:09.970996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:09.971021 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:09.971433 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.971445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.972993 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.972998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.975271 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.975288 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.977131 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.977138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.977334 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.977378 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 21:34:09.977541 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 21:34:09.977737 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2013,"thread_start_us":8675,"threads_started":1}
I20260926 21:34:09.979404 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3ad5f3352f4080aaeb92408affe2bc" candidate_uuid: "0b3ba16deddf47708f84d5f056886076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e1183b0b14cb1bf5d634442b938b7" is_pre_election: true
I20260926 21:34:09.979929 16571 raft_consensus.cc:2454] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b3ba16deddf47708f84d5f056886076 in term 0.
I20260926 21:34:09.980365 16879 heartbeater.cc:503] Master 127.0.97.62:33813 was elected leader, sending a full tablet report...
I20260926 21:34:09.980664 16636 leader_election.cc:302] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b3ba16deddf47708f84d5f056886076, f65e1183b0b14cb1bf5d634442b938b7; no voters:
I20260926 21:34:09.981108 16901 raft_consensus.cc:2781] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:34:09.981240 16899 ts_tablet_manager.cc:1428] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: Time spent starting tablet: real 0.020s user 0.012s sys 0.004s
I20260926 21:34:09.981283 16901 raft_consensus.cc:482] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:34:09.981393 16901 raft_consensus.cc:3037] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:09.984506 16901 raft_consensus.cc:504] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.985354 16901 leader_election.cc:288] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [CANDIDATE]: Term 1 election: Requested vote from peers f65e1183b0b14cb1bf5d634442b938b7 (127.0.97.1:41671), 299b2665d202496f907e8b05ab3658c9 (127.0.97.3:37583)
I20260926 21:34:09.985749 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3ad5f3352f4080aaeb92408affe2bc" candidate_uuid: "0b3ba16deddf47708f84d5f056886076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65e1183b0b14cb1bf5d634442b938b7"
I20260926 21:34:09.985859 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.969092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45455 (local address 127.0.97.3:37583)
0926 21:34:09.969351 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:09.969360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:09.977248 (+ 7888us) server_negotiation.cc:407] Connection header received
0926 21:34:09.977494 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:09.977498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:09.977599 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:09.977685 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:09.978839 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.980870 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.983488 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.984799 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.984925 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.985032 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:09.985458 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 21:34:09.985598 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:34:09.985966 16571 raft_consensus.cc:3037] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:09.986500 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:09.968249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37583 (local address 127.0.97.2:45455)
0926 21:34:09.977118 (+ 8869us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:09.977138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:09.977299 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:09.977838 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:09.977844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:09.977869 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:09.978599 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.978612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.981059 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.981065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.983225 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:09.983244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:09.985022 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:09.985029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:09.985257 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:09.985303 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 21:34:09.985952 (+ 649us) client_negotiation.cc:240] Negotiation successful
0926 21:34:09.986110 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8714,"thread_start_us":1750,"threads_started":1}
I20260926 21:34:09.987514 16834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3ad5f3352f4080aaeb92408affe2bc" candidate_uuid: "0b3ba16deddf47708f84d5f056886076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299b2665d202496f907e8b05ab3658c9" is_pre_election: true
I20260926 21:34:09.987555 16833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3ad5f3352f4080aaeb92408affe2bc" candidate_uuid: "0b3ba16deddf47708f84d5f056886076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299b2665d202496f907e8b05ab3658c9"
I20260926 21:34:09.988008 16834 raft_consensus.cc:2454] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b3ba16deddf47708f84d5f056886076 in term 0.
I20260926 21:34:09.989106 16571 raft_consensus.cc:2454] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b3ba16deddf47708f84d5f056886076 in term 1.
I20260926 21:34:09.989538 16636 leader_election.cc:302] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b3ba16deddf47708f84d5f056886076, f65e1183b0b14cb1bf5d634442b938b7; no voters: 299b2665d202496f907e8b05ab3658c9
I20260926 21:34:09.989892 16901 raft_consensus.cc:2781] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:34:09.990092 16901 raft_consensus.cc:686] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 1 LEADER]: Becoming Leader. State: Replica: 0b3ba16deddf47708f84d5f056886076, State: Running, Role: LEADER
I20260926 21:34:09.990621 16901 consensus_queue.cc:237] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } }
I20260926 21:34:09.995576 16431 catalog_manager.cc:5495] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b3ba16deddf47708f84d5f056886076 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0b3ba16deddf47708f84d5f056886076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b3ba16deddf47708f84d5f056886076" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 } health_report { overall_health: UNKNOWN } } }
I20260926 21:34:10.017278 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.012022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.0.97.2:40103)
0926 21:34:10.012217 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:10.012221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:10.012256 (+ 35us) server_negotiation.cc:407] Connection header received
0926 21:34:10.012433 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:10.012437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:10.012515 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:10.012641 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:34:10.013009 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:10.013818 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:10.015190 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:10.015583 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:10.015675 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:10.016087 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:34:10.016692 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:34:10.016804 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:10.016993 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 21:34:10.017059 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:34:10.025740 16833 raft_consensus.cc:3037] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:10.026046 16571 raft_consensus.cc:1261] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Refusing update from remote peer 0b3ba16deddf47708f84d5f056886076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:34:10.027043 16903 consensus_queue.cc:1035] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65e1183b0b14cb1bf5d634442b938b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:34:10.028746 16833 raft_consensus.cc:1261] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [term 1 FOLLOWER]: Refusing update from remote peer 0b3ba16deddf47708f84d5f056886076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:34:10.029601 16903 consensus_queue.cc:1035] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299b2665d202496f907e8b05ab3658c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:34:10.051141 16431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:34:10.051364 16431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:34:10.055009 16431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Sending DeleteTablet for 3 replicas of tablet 2e3ad5f3352f4080aaeb92408affe2bc
I20260926 21:34:10.056254 16683 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3ad5f3352f4080aaeb92408affe2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42902
I20260926 21:34:10.056265 16551 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3ad5f3352f4080aaeb92408affe2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39550
I20260926 21:34:10.056401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 16468
I20260926 21:34:10.056528 16814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3ad5f3352f4080aaeb92408affe2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39984
I20260926 21:34:10.067373 16919 tablet_replica.cc:330] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: stopping tablet replica
I20260926 21:34:10.068205 16919 raft_consensus.cc:2229] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:34:10.069072 16919 raft_consensus.cc:2258] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:10.069686 16920 tablet_replica.cc:330] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: stopping tablet replica
I20260926 21:34:10.069875 16918 tablet_replica.cc:330] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: stopping tablet replica
I20260926 21:34:10.070160 16920 raft_consensus.cc:2229] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:34:10.070714 16920 raft_consensus.cc:2258] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:10.070674 16918 raft_consensus.cc:2229] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:34:10.072999 16919 ts_tablet_manager.cc:1905] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:10.074173 16920 ts_tablet_manager.cc:1905] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:10.075088 16918 raft_consensus.cc:2258] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:10.076097 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.057708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.0.97.1:41671)
0926 21:34:10.057854 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:10.057860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:10.057877 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:34:10.057933 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:10.057938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:10.058020 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:10.058115 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:10.060055 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:10.061250 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:10.063210 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:10.063909 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:10.064043 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:10.068013 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:10.068033 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:10.068052 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:10.068105 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:10.072539 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:10.073506 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:10.073516 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:10.073520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:10.073681 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:10.074137 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:10.074143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:10.074147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:10.074779 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:10.075108 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:10.075504 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 21:34:10.075736 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 21:34:10.076987 16611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:10.077456 16918 ts_tablet_manager.cc:1905] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:10.082337 16920 ts_tablet_manager.cc:1918] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:10.082545 16920 log.cc:1196] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/2e3ad5f3352f4080aaeb92408affe2bc
I20260926 21:34:10.083088 16920 ts_tablet_manager.cc:1939] T 2e3ad5f3352f4080aaeb92408affe2bc P 299b2665d202496f907e8b05ab3658c9: Deleting consensus metadata
I20260926 21:34:10.084867 16918 ts_tablet_manager.cc:1918] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:10.084955 16418 catalog_manager.cc:4841] TS 299b2665d202496f907e8b05ab3658c9 (127.0.97.3:37583): tablet 2e3ad5f3352f4080aaeb92408affe2bc (table test-table [id=46416b5344d04bf1b39eb7396957e6ba]) successfully deleted
I20260926 21:34:10.085057 16918 log.cc:1196] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/2e3ad5f3352f4080aaeb92408affe2bc
I20260926 21:34:10.085525 16918 ts_tablet_manager.cc:1939] T 2e3ad5f3352f4080aaeb92408affe2bc P 0b3ba16deddf47708f84d5f056886076: Deleting consensus metadata
I20260926 21:34:10.087153 16418 catalog_manager.cc:4841] TS 0b3ba16deddf47708f84d5f056886076 (127.0.97.2:40103): tablet 2e3ad5f3352f4080aaeb92408affe2bc (table test-table [id=46416b5344d04bf1b39eb7396957e6ba]) successfully deleted
I20260926 21:34:10.182638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 16468
W20260926 21:34:10.200845 16415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41671 (error 108)
I20260926 21:34:10.201644 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 16620
I20260926 21:34:10.203137 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.202557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39568)
0926 21:34:10.202843 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:10.202958 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260926 21:34:10.203418 16415 catalog_manager.cc:4584] TS f65e1183b0b14cb1bf5d634442b938b7 (127.0.97.1:41671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e3ad5f3352f4080aaeb92408affe2bc: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
I20260926 21:34:10.222704 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.202538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.0.97.2:40103)
0926 21:34:10.202726 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:10.202733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:10.202886 (+ 153us) server_negotiation.cc:407] Connection header received
0926 21:34:10.203107 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:10.203112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:10.203173 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:10.203309 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:10.204806 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:10.205674 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:10.207103 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:10.207546 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:10.207704 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:10.214805 (+ 7101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:10.214827 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:10.214850 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:10.214905 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:10.219195 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:10.220127 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:10.220198 (+ 71us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:10.220202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:10.220408 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:10.220881 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:10.220886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:10.220890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:10.221391 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:10.221740 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:10.222006 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 21:34:10.222242 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:34:10.223632 16743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:10.257323 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.256836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39576)
0926 21:34:10.257055 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:10.257163 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 21:34:10.258396 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.258036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39584)
0926 21:34:10.258198 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:10.258277 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 21:34:10.319155 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.318656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39600)
0926 21:34:10.318887 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:10.318985 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 21:34:10.320209 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.319840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39602)
0926 21:34:10.320014 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:10.320085 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:34:10.338678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 16620
I20260926 21:34:10.357537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 16751
I20260926 21:34:10.372632 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.358320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.0.97.3:37583)
0926 21:34:10.358600 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:10.358604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:10.358623 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:34:10.358793 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:10.358797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:10.358866 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:10.358956 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:10.360560 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:10.361294 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:10.363096 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:10.363505 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:10.363600 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:10.366846 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:10.366860 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:10.366876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:10.366909 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:10.369757 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:10.370641 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:10.370648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:10.370651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:10.370777 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:10.371192 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:10.371196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:10.371198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:10.371547 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:10.371748 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:10.372085 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 21:34:10.372262 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260926 21:34:10.373308 16874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:10.400642 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.400178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39606)
0926 21:34:10.400405 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:10.400491 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 21:34:10.401772 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.401350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39608)
0926 21:34:10.401568 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:10.401662 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260926 21:34:10.464397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 16751
I20260926 21:34:10.481628 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 16398
I20260926 21:34:10.482781 16461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:10.599308 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:10.598411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41671 (local address 127.0.0.1:39622)
0926 21:34:10.598784 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:10.598914 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260926 21:34:10.599821 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 16398
I20260926 21:34:30.622769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33813
--webserver_interface=127.0.97.62
--webserver_port=45361
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:30.982139 16926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:30.982511 16926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:30.982688 16926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:30.990306 16926 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 21:34:30.990410 16926 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 21:34:30.990527 16926 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 21:34:30.990629 16926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:34:30.990720 16926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:30.990847 16926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:30.990926 16926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:34:30.991006 16926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:34:31.001199 16926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:31.002620 16926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:31.004066 16926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:31.017238 16931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:31.017243 16934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:31.017144 16932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:31.017829 16926 server_base.cc:1034] running on GCE node
I20260926 21:34:31.018802 16926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:31.020427 16926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:31.021718 16926 hybrid_clock.cc:648] HybridClock initialized: now 1790458471021632 us; error 39 us; skew 500 ppm
I20260926 21:34:31.022207 16926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:31 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:31.030500 16926 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:34:31.032662 16926 webserver.cc:460] Webserver started at http://127.0.97.62:45361/ using document root <none> and password file <none>
I20260926 21:34:31.033648 16926 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:31.033844 16926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:31.041095 16926 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:34:31.044795 16941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:31.046496 16926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:34:31.046727 16926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418"
format_stamp: "Formatted at 2026-09-26 21:33:44 on dist-test-slave-87bt"
server_key: "2ec9910a262600d7fb48af790c1f9316"
server_key_iv: "dc61022b3fb3e537bc4c67369b652aaa"
server_key_version: "encryptionkey@0"
I20260926 21:34:31.047899 16926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:31.073423 16926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:31.078013 16926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:31.078316 16926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:31.098220 16926 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33813
I20260926 21:34:31.098341 16992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33813 every 8 connection(s)
I20260926 21:34:31.100008 16926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:34:31.107972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 16926
I20260926 21:34:31.108551 16993 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:34:31.109148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41671
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33813
--webserver_port=37733
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:34:31.114705 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Bootstrap starting.
I20260926 21:34:31.139724 16993 log.cc:824] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:31.157819 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:34:31.158746 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Bootstrap complete.
I20260926 21:34:31.169222 16993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:34:31.170383 16993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4e2cf1e9a04597b80c44e5a4fd6418, State: Initialized, Role: FOLLOWER
I20260926 21:34:31.171223 16993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:34:31.171506 16993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:34:31.171671 16993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:34:31.171887 16993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 2 FOLLOWER]: Advancing to term 3
I20260926 21:34:31.176242 16993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:34:31.176892 16993 leader_election.cc:302] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a4e2cf1e9a04597b80c44e5a4fd6418; no voters:
I20260926 21:34:31.177668 16993 leader_election.cc:288] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 21:34:31.177827 16997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 21:34:31.179131 16997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [term 3 LEADER]: Becoming Leader. State: Replica: 3a4e2cf1e9a04597b80c44e5a4fd6418, State: Running, Role: LEADER
I20260926 21:34:31.179790 16997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } }
I20260926 21:34:31.180336 16993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:34:31.182778 16998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } } }
I20260926 21:34:31.183276 16998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:31.187254 16999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4e2cf1e9a04597b80c44e5a4fd6418. Latest consensus state: current_term: 3 leader_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4e2cf1e9a04597b80c44e5a4fd6418" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33813 } } }
I20260926 21:34:31.187709 16999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418 [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:31.189726 17003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:34:31.202174 17003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:34:31.204217 17003 catalog_manager.cc:1259] Loaded cluster ID: 92aec389c7734cf8a53fd3949cde9d32
I20260926 21:34:31.204397 17003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:34:31.210933 17003 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:34:31.213510 17003 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Loaded TSK: 0
I20260926 21:34:31.213732 17003 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a4e2cf1e9a04597b80c44e5a4fd6418: Loaded TSK: 1
F20260926 21:34:31.215461 17003 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7552c3f9ad google::LogMessage::Fail() at ??:0
@ 0x7f7552c4288a google::LogMessage::SendToLog() at ??:0
@ 0x7f7552c3f4de google::LogMessage::Flush() at ??:0
@ 0x7f7552c40a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f75569a37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f755699273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f756866e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f756866d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f75686f92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f75686f90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f75687435e9 std::function<>::operator()() at ??:0
@ 0x7f7568671cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f756866fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f75686f7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f75686f7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f755588b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f755588e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f755588e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7555869eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7558b436db start_thread at ??:0
@ 0x7f7551ad961f clone at ??:0
W20260926 21:34:31.512319 16995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:31.512681 16995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:31.512836 16995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:31.520226 16995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:31.520339 16995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:31.520531 16995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:34:31.530812 16995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37733
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:31.532210 16995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:31.533533 16995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:31.546710 17040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:31.546689 17041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:31.546976 17043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:31.547622 16995 server_base.cc:1034] running on GCE node
I20260926 21:34:31.548295 16995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:31.549336 16995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:31.550531 16995 hybrid_clock.cc:648] HybridClock initialized: now 1790458471550455 us; error 66 us; skew 500 ppm
I20260926 21:34:31.551079 16995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:31 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:31.554399 16995 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:34:31.555989 16995 webserver.cc:460] Webserver started at http://127.0.97.1:37733/ using document root <none> and password file <none>
I20260926 21:34:31.556661 16995 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:31.556766 16995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:31.562235 16995 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 21:34:31.565173 17050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:31.566581 16995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:34:31.566749 16995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "f65e1183b0b14cb1bf5d634442b938b7"
format_stamp: "Formatted at 2026-09-26 21:33:45 on dist-test-slave-87bt"
server_key: "ee644f31753f624606e25e37296d635b"
server_key_iv: "4bbe23df0397f7b503c5e7c448b65105"
server_key_version: "encryptionkey@0"
I20260926 21:34:31.567718 16995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:31.585690 16995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:31.590119 16995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:31.590401 16995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:31.591562 16995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:31.593293 17057 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 21:34:31.598605 17057 ts_tablet_manager.cc:1852] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 21:34:31.599566 17057 ts_tablet_manager.cc:1905] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:31.606693 17057 ts_tablet_manager.cc:1918] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:31.606886 17057 log.cc:1196] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/2e3ad5f3352f4080aaeb92408affe2bc
I20260926 21:34:31.607386 17057 ts_tablet_manager.cc:1939] T 2e3ad5f3352f4080aaeb92408affe2bc P f65e1183b0b14cb1bf5d634442b938b7: Deleting consensus metadata
I20260926 21:34:31.607720 16995 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 21:34:31.607853 16995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260926 21:34:31.608006 16995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:31.608098 16995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:31.642405 16995 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41671
I20260926 21:34:31.642519 17163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41671 every 8 connection(s)
I20260926 21:34:31.643865 16995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:34:31.648253 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 16995
I20260926 21:34:31.649099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40103
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33813
--webserver_port=33867
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:34:31.649852 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:31.647234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:50431)
0926 21:34:31.648210 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:31.648504 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":368,"threads_started":1}
W20260926 21:34:31.650604 17164 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33813 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33813: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
I20260926 21:34:31.651652 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:31.651173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:40457)
0926 21:34:31.651377 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:31.651491 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 21:34:31.652663 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:31.652232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:46225)
0926 21:34:31.652395 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:31.652502 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:34:31.652899 17164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:34:31.983637 17167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:31.983930 17167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:31.984095 17167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:31.991575 17167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:31.991688 17167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:31.991879 17167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:34:32.001819 17167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33867
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:32.003185 17167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:32.004585 17167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:32.018214 17172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:32.018185 17173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:32.018311 17175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:32.019574 17167 server_base.cc:1034] running on GCE node
I20260926 21:34:32.020216 17167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:32.021186 17167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:32.022387 17167 hybrid_clock.cc:648] HybridClock initialized: now 1790458472022339 us; error 36 us; skew 500 ppm
I20260926 21:34:32.022758 17167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:32 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:32.025878 17167 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:34:32.027420 17167 webserver.cc:460] Webserver started at http://127.0.97.2:33867/ using document root <none> and password file <none>
I20260926 21:34:32.028117 17167 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:32.028296 17167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:32.033723 17167 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 21:34:32.036480 17182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:32.037788 17167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:34:32.037973 17167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "0b3ba16deddf47708f84d5f056886076"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "0082740178576175b394ce1207d8ad1f"
server_key_iv: "c6f46c8d54c453b6086ea4fd81beff35"
server_key_version: "encryptionkey@0"
I20260926 21:34:32.039103 17167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:32.075202 17167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:32.079313 17167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:32.079598 17167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:32.080606 17167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:32.082031 17167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:32.082156 17167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:32.082283 17167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:32.082369 17167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:32.116133 17167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40103
I20260926 21:34:32.116226 17294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40103 every 8 connection(s)
I20260926 21:34:32.117628 17167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:34:32.122582 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:32.120676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:40159)
0926 21:34:32.121417 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:32.121626 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":278,"threads_started":1}
W20260926 21:34:32.123232 17295 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33813 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33813: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
I20260926 21:34:32.123944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 17167
I20260926 21:34:32.124105 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:32.123687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:34851)
0926 21:34:32.123873 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:32.123962 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 21:34:32.124727 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37583
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33813
--webserver_port=44189
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:34:32.125094 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:32.124682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:36061)
0926 21:34:32.124837 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:32.124943 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 21:34:32.125352 17295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:34:32.456112 17298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:32.456411 17298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:32.456573 17298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:32.463902 17298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:32.464040 17298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:32.464223 17298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:34:32.474437 17298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44189
--tserver_master_addrs=127.0.97.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:32.475795 17298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:32.477209 17298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:32.490455 17303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:32.490404 17306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:32.490371 17304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:32.491359 17298 server_base.cc:1034] running on GCE node
I20260926 21:34:32.492014 17298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:32.492973 17298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:32.494267 17298 hybrid_clock.cc:648] HybridClock initialized: now 1790458472494200 us; error 51 us; skew 500 ppm
I20260926 21:34:32.494681 17298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:32 dist-test-slave-87bt krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:32.498204 17298 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:34:32.499899 17298 webserver.cc:460] Webserver started at http://127.0.97.3:44189/ using document root <none> and password file <none>
I20260926 21:34:32.500669 17298 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:32.500819 17298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:32.506660 17298 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 21:34:32.509668 17313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:32.511278 17298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:34:32.511482 17298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "299b2665d202496f907e8b05ab3658c9"
format_stamp: "Formatted at 2026-09-26 21:33:46 on dist-test-slave-87bt"
server_key: "54c4d3711d2a647e6365040d27f400dd"
server_key_iv: "1ee6671770a12de2e9174d6e3c50b43e"
server_key_version: "encryptionkey@0"
I20260926 21:34:32.512593 17298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:32.525060 17298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:32.529824 17298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:32.530117 17298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:32.531271 17298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:32.532688 17298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:32.532824 17298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:32.532958 17298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:32.533033 17298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:32.566808 17298 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37583
I20260926 21:34:32.566900 17425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37583 every 8 connection(s)
I20260926 21:34:32.568274 17298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:34:32.569034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 17298
I20260926 21:34:32.574224 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:32.572398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:54225)
0926 21:34:32.573144 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:32.573334 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":292,"threads_started":1}
W20260926 21:34:32.574940 17426 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33813 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33813: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
I20260926 21:34:32.575753 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:32.575448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:39459)
0926 21:34:32.575585 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:32.575651 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 21:34:32.576545 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:32.576250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:54821)
0926 21:34:32.576365 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:32.576441 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:34:32.576905 17426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 21:34:32.654206 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:32.653453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:47639)
0926 21:34:32.653876 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:32.653977 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":203,"threads_started":1}
I20260926 21:34:33.126893 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:33.125970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:43077)
0926 21:34:33.126546 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:33.126661 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":252,"threads_started":1}
I20260926 21:34:33.578159 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:33.577384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:56647)
0926 21:34:33.577839 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:33.577932 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260926 21:34:33.655781 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:33.655124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:39383)
0926 21:34:33.655539 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:33.655619 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":214,"threads_started":1}
I20260926 21:34:34.128589 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:34.127877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:45841)
0926 21:34:34.128330 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:34.128411 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":215,"threads_started":1}
I20260926 21:34:34.579774 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:34.579101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:44885)
0926 21:34:34.579539 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:34.579616 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":210,"threads_started":1}
I20260926 21:34:34.657733 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:34.656938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:34775)
0926 21:34:34.657465 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:34.657560 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":263,"threads_started":1}
I20260926 21:34:35.130131 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:35.129402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:37977)
0926 21:34:35.129877 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:35.129961 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":241,"threads_started":1}
I20260926 21:34:35.581421 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:35.580618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:58433)
0926 21:34:35.581164 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:35.581246 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":312,"threads_started":1}
I20260926 21:34:35.659510 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:35.658799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:55991)
0926 21:34:35.659258 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:35.659340 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":246,"threads_started":1}
I20260926 21:34:36.131893 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:36.131157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:43751)
0926 21:34:36.131636 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:36.131720 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":241,"threads_started":1}
I20260926 21:34:36.582968 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:36.582239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:45571)
0926 21:34:36.582686 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:36.582757 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":252,"threads_started":1}
I20260926 21:34:36.661047 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:36.660364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:33251)
0926 21:34:36.660797 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:36.660874 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":210,"threads_started":1}
I20260926 21:34:37.133589 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:37.132899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:37121)
0926 21:34:37.133330 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:37.133414 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20260926 21:34:37.584525 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:37.583843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:51737)
0926 21:34:37.584282 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:37.584359 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260926 21:34:37.662760 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:37.661990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:44235)
0926 21:34:37.662517 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:37.662603 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":240,"threads_started":1}
I20260926 21:34:38.135401 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:38.134572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:59027)
0926 21:34:38.135054 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:38.135231 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
I20260926 21:34:38.586354 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:38.585601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:57657)
0926 21:34:38.586078 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:38.586176 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260926 21:34:38.664511 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:38.663701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:37805)
0926 21:34:38.664213 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:38.664306 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":250,"threads_started":1}
I20260926 21:34:39.137428 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:39.136389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:54597)
0926 21:34:39.137033 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:39.137152 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":301,"threads_started":1}
I20260926 21:34:39.588151 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:39.587271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:35329)
0926 21:34:39.587807 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:39.587897 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":258,"threads_started":1}
I20260926 21:34:39.666096 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:39.665425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:43497)
0926 21:34:39.665846 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:39.665933 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260926 21:34:40.139269 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:40.138577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:39291)
0926 21:34:40.139027 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:40.139110 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":216,"threads_started":1}
I20260926 21:34:40.589781 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:40.589078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:57417)
0926 21:34:40.589548 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:40.589625 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260926 21:34:40.667724 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:40.666987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:56405)
0926 21:34:40.667455 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:40.667557 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":232,"threads_started":1}
I20260926 21:34:41.140956 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:41.140218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:56235)
0926 21:34:41.140695 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:41.140779 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":237,"threads_started":1}
I20260926 21:34:41.591436 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:41.590725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:56979)
0926 21:34:41.591162 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:41.591235 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":206,"threads_started":1}
I20260926 21:34:41.669466 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:41.668799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:60015)
0926 21:34:41.669227 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:41.669303 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":231,"threads_started":1}
I20260926 21:34:42.142804 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:42.142013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:41403)
0926 21:34:42.142493 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:42.142629 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":250,"threads_started":1}
I20260926 21:34:42.593140 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:42.592437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:39081)
0926 21:34:42.592909 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:42.592989 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
I20260926 21:34:42.671501 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:42.670583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:33911)
0926 21:34:42.671090 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:42.671198 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":292,"threads_started":1}
I20260926 21:34:43.144475 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:43.143745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:48169)
0926 21:34:43.144204 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:43.144292 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":206,"threads_started":1}
I20260926 21:34:43.595098 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:43.594257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:32857)
0926 21:34:43.594832 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:43.594921 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":321,"threads_started":1}
I20260926 21:34:43.673383 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:43.672583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:40625)
0926 21:34:43.673113 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:43.673213 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":282,"threads_started":1}
I20260926 21:34:44.146226 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:44.145529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:46395)
0926 21:34:44.145984 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:44.146068 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260926 21:34:44.596869 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:44.596150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:33547)
0926 21:34:44.596601 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:44.596684 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
I20260926 21:34:44.675179 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:44.674388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:39351)
0926 21:34:44.674896 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:44.674977 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":227,"threads_started":1}
I20260926 21:34:45.147962 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:45.147170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:60465)
0926 21:34:45.147679 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:45.147771 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":243,"threads_started":1}
I20260926 21:34:45.598627 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:45.597845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:36261)
0926 21:34:45.598314 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:45.598396 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":225,"threads_started":1}
I20260926 21:34:45.676818 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:45.676081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:39455)
0926 21:34:45.676550 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:45.676635 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260926 21:34:46.149657 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:46.148952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:34795)
0926 21:34:46.149392 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:46.149476 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
I20260926 21:34:46.600246 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:46.599554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:44953)
0926 21:34:46.599995 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:46.600074 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":213,"threads_started":1}
I20260926 21:34:46.678840 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:46.677971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:43545)
0926 21:34:46.678538 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:46.678657 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260926 21:34:47.151603 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:47.150795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.2:55319)
0926 21:34:47.151316 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:47.151400 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":282,"threads_started":1}
I20260926 21:34:47.576056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 16995
I20260926 21:34:47.600041 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:47.577348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.0.97.1:41671)
0926 21:34:47.577960 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:47.577974 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:47.578011 (+ 37us) server_negotiation.cc:407] Connection header received
0926 21:34:47.579752 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:47.579764 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:47.580212 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:47.580582 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:47.582119 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:47.583498 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:47.584805 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:47.585488 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:47.585690 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:47.593598 (+ 7908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:47.593607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:47.593626 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:47.593669 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:47.596497 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:47.597270 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:47.597278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:47.597286 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:47.597458 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:47.597856 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:47.597861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:47.597864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:47.598495 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:47.598831 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:47.599266 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 21:34:47.599740 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":286,"threads_started":1}
I20260926 21:34:47.601575 17159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:47.602254 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:47.601295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.3:34927)
0926 21:34:47.601819 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:47.601974 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":231,"threads_started":1}
I20260926 21:34:47.689236 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:47.688219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33813 (local address 127.0.97.1:41867)
0926 21:34:47.688803 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:47.688951 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":290,"threads_started":1}
I20260926 21:34:47.689561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 16995
I20260926 21:34:47.707082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 17167
I20260926 21:34:47.725219 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:47.708036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.0.97.2:40103)
0926 21:34:47.708537 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:47.708551 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:47.708587 (+ 36us) server_negotiation.cc:407] Connection header received
0926 21:34:47.710109 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:47.710117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:47.710626 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:47.710982 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:47.712336 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:47.713530 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:47.714917 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:47.715585 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:47.715783 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:47.718736 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:47.718745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:47.718761 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:47.718800 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:47.721670 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:47.722473 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:47.722486 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:47.722493 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:47.722676 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:47.723069 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:47.723075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:47.723078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:47.723639 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:47.723957 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:47.724471 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 21:34:47.724923 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260926 21:34:47.726768 17290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:47.813505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 17167
I20260926 21:34:47.831336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 17298
I20260926 21:34:47.849823 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:47.832276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.0.97.3:37583)
0926 21:34:47.832952 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:47.832965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:47.833006 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:34:47.834805 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:47.834817 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:47.835272 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:47.835637 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:47.836990 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:47.838226 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:47.839726 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:47.840405 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:47.840540 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:47.843381 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:47.843388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:47.843404 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:47.843447 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:47.846292 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:47.847305 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:47.847313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:47.847319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:47.847483 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:47.847920 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:47.847923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:47.847925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:47.848486 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:47.848712 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:47.849160 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 21:34:47.849526 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":359,"threads_started":1}
I20260926 21:34:47.851351 17421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:47.934719 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 17298
2026-09-26T21:34:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64752 ms)
[----------] 30 tests from SecurityITest (300747 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:34:48 dist-test-slave-87bt krb5kdc[17484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:34:48 dist-test-slave-87bt krb5kdc[17484](info): set up 2 sockets
Sep 26 21:34:48 dist-test-slave-87bt krb5kdc[17484](info): commencing operation
krb5kdc: starting...
W20260926 21:34:49.034652 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:34:49 dist-test-slave-87bt krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:34:49.282033 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42339
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:49.621874 17497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:49.622273 17497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:49.622370 17497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:49.629897 17497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:34:49.630017 17497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:49.630062 17497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:49.630111 17497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:34:49.630156 17497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:34:49.640383 17497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:49.641665 17497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:49.642974 17497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:49.655488 17502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:49.655380 17505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:49.655421 17503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:49.655933 17497 server_base.cc:1034] running on GCE node
I20260926 21:34:49.656955 17497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:49.658489 17497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:49.659789 17497 hybrid_clock.cc:648] HybridClock initialized: now 1790458489659723 us; error 45 us; skew 500 ppm
I20260926 21:34:49.660208 17497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:49 dist-test-slave-87bt krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:49.664104 17497 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:34:49.667693 17497 webserver.cc:460] Webserver started at http://127.0.97.62:45285/ using document root <none> and password file <none>
I20260926 21:34:49.668697 17497 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:49.668839 17497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:49.669239 17497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:49.673888 17497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "fff8b86d8f73430e8be39b5e65d79dec"
format_stamp: "Formatted at 2026-09-26 21:34:49 on dist-test-slave-87bt"
server_key: "74cb192a94550c336ec298f170b58f02"
server_key_iv: "fa44f0cd3aa3159df98b9426830ebd74"
server_key_version: "encryptionkey@0"
I20260926 21:34:49.675113 17497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "fff8b86d8f73430e8be39b5e65d79dec"
format_stamp: "Formatted at 2026-09-26 21:34:49 on dist-test-slave-87bt"
server_key: "74cb192a94550c336ec298f170b58f02"
server_key_iv: "fa44f0cd3aa3159df98b9426830ebd74"
server_key_version: "encryptionkey@0"
I20260926 21:34:49.684437 17497 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260926 21:34:49.689621 17512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:49.691633 17497 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 21:34:49.691881 17497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "fff8b86d8f73430e8be39b5e65d79dec"
format_stamp: "Formatted at 2026-09-26 21:34:49 on dist-test-slave-87bt"
server_key: "74cb192a94550c336ec298f170b58f02"
server_key_iv: "fa44f0cd3aa3159df98b9426830ebd74"
server_key_version: "encryptionkey@0"
I20260926 21:34:49.692140 17497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:49.711027 17497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:49.716311 17497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:49.716614 17497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:49.735435 17497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42339
I20260926 21:34:49.735456 17563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42339 every 8 connection(s)
I20260926 21:34:49.736929 17497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:34:49.737285 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 17497
I20260926 21:34:49.737498 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:34:49.738634 388 external_mini_cluster.cc:1428] Setting key 5ee13300be7f261944e8b2db5a9fa528
I20260926 21:34:49.741815 17564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:34:49 dist-test-slave-87bt krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:49.754402 17564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec: Bootstrap starting.
I20260926 21:34:49.758507 17564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:49.759855 17564 log.cc:824] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:49.763320 17564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec: No bootstrap required, opened a new log
I20260926 21:34:49.764381 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:49.740753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.97.62:42339)
0926 21:34:49.741533 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:49.741548 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:49.741600 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:34:49.743940 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:49.743967 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:49.744449 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:49.744899 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:49.746263 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:49.748419 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:49.749880 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:49.750916 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:49.751242 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:49.755608 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:49.755618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:49.755637 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:49.755683 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:49.759604 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:49.760516 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:49.760525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:49.760533 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:49.760706 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:49.761144 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:49.761147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:49.761149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:49.761847 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:49.762118 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:49.762539 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 21:34:49.762970 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":322,"threads_started":1}
I20260926 21:34:49.776166 17564 raft_consensus.cc:348] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff8b86d8f73430e8be39b5e65d79dec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42339 } }
I20260926 21:34:49.776573 17564 raft_consensus.cc:374] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:49.776684 17564 raft_consensus.cc:729] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff8b86d8f73430e8be39b5e65d79dec, State: Initialized, Role: FOLLOWER
I20260926 21:34:49.777348 17564 consensus_queue.cc:260] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff8b86d8f73430e8be39b5e65d79dec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42339 } }
I20260926 21:34:49.777597 17564 raft_consensus.cc:388] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:34:49.777741 17564 raft_consensus.cc:482] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:34:49.777912 17564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:49.780869 17564 raft_consensus.cc:504] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff8b86d8f73430e8be39b5e65d79dec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42339 } }
I20260926 21:34:49.781459 17564 leader_election.cc:302] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fff8b86d8f73430e8be39b5e65d79dec; no voters:
I20260926 21:34:49.782140 17564 leader_election.cc:288] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:34:49.782311 17569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:34:49.783576 17569 raft_consensus.cc:686] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [term 1 LEADER]: Becoming Leader. State: Replica: fff8b86d8f73430e8be39b5e65d79dec, State: Running, Role: LEADER
I20260926 21:34:49.784085 17569 consensus_queue.cc:237] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff8b86d8f73430e8be39b5e65d79dec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42339 } }
I20260926 21:34:49.784405 17564 sys_catalog.cc:564] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:34:49.787012 17570 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fff8b86d8f73430e8be39b5e65d79dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff8b86d8f73430e8be39b5e65d79dec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42339 } } }
I20260926 21:34:49.786986 17571 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader fff8b86d8f73430e8be39b5e65d79dec. Latest consensus state: current_term: 1 leader_uuid: "fff8b86d8f73430e8be39b5e65d79dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff8b86d8f73430e8be39b5e65d79dec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42339 } } }
I20260926 21:34:49.787480 17570 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:49.787480 17571 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:49.788430 17575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:34:49.801685 17575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:34:49.812520 17575 catalog_manager.cc:1347] Generated new cluster ID: 706b0513bfcc48c19da6c05fcc090356
I20260926 21:34:49.812664 17575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:34:49.823383 17575 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:34:49.824862 17575 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:34:49.832880 17575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec: Generated new TSK 0
I20260926 21:34:49.833767 17575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:34:49.901342 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42339
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:50.263659 17590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:50.263945 17590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:50.264138 17590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:50.271651 17590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:50.271764 17590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:50.271975 17590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:34:50.281700 17590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:50.283099 17590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:50.284492 17590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:50.298785 17595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:50.298808 17598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:50.298748 17596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:50.299476 17590 server_base.cc:1034] running on GCE node
I20260926 21:34:50.300040 17590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:50.301003 17590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:50.302198 17590 hybrid_clock.cc:648] HybridClock initialized: now 1790458490302117 us; error 81 us; skew 500 ppm
I20260926 21:34:50.302587 17590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:50 dist-test-slave-87bt krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:50.305639 17590 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:34:50.307137 17590 webserver.cc:460] Webserver started at http://127.0.97.1:40697/ using document root <none> and password file <none>
I20260926 21:34:50.307952 17590 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:50.308096 17590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:50.308408 17590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:50.311317 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "2b253f06d1144297b2cf57c315ce3a29"
format_stamp: "Formatted at 2026-09-26 21:34:50 on dist-test-slave-87bt"
server_key: "34a52b717441e43d036d07ca3d915d56"
server_key_iv: "bb775b03d1be1bd58242c1acc607cad4"
server_key_version: "encryptionkey@0"
I20260926 21:34:50.312201 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2b253f06d1144297b2cf57c315ce3a29"
format_stamp: "Formatted at 2026-09-26 21:34:50 on dist-test-slave-87bt"
server_key: "34a52b717441e43d036d07ca3d915d56"
server_key_iv: "bb775b03d1be1bd58242c1acc607cad4"
server_key_version: "encryptionkey@0"
I20260926 21:34:50.317237 17590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:34:50.321070 17605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:50.322378 17590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 21:34:50.322595 17590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "2b253f06d1144297b2cf57c315ce3a29"
format_stamp: "Formatted at 2026-09-26 21:34:50 on dist-test-slave-87bt"
server_key: "34a52b717441e43d036d07ca3d915d56"
server_key_iv: "bb775b03d1be1bd58242c1acc607cad4"
server_key_version: "encryptionkey@0"
I20260926 21:34:50.322824 17590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:50.348907 17590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:50.353080 17590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:50.353382 17590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:50.354362 17590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:50.355719 17590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:50.355823 17590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:50.355970 17590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:50.356045 17590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:50.388841 17590 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42513
I20260926 21:34:50.388921 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42513 every 8 connection(s)
I20260926 21:34:50.390380 17590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:34:50.400226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 17590
I20260926 21:34:50.400440 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:34:50.401196 388 external_mini_cluster.cc:1428] Setting key 1e8f015b5e6bce1729472de017bb777c
Sep 26 21:34:50 dist-test-slave-87bt krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:50.411217 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:50.392911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53523 (local address 127.0.97.62:42339)
0926 21:34:50.393395 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:50.393402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:50.394383 (+ 981us) server_negotiation.cc:407] Connection header received
0926 21:34:50.396268 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:50.396272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:50.396341 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:50.396439 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:50.398095 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:50.398898 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:50.400840 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:50.401224 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:50.401369 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:50.405188 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:50.405195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:50.405202 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:50.405267 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:50.408123 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:50.408856 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:50.408863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:50.408867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:50.408997 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:50.409430 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:50.409436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:50.409439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:50.409714 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:50.409870 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:50.410623 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 21:34:50.410860 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":222,"threads_started":1}
I20260926 21:34:50.412253 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:50.393499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42339 (local address 127.0.97.1:53523)
0926 21:34:50.394214 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:50.394258 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:50.396008 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:50.396710 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:50.396719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:50.397265 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:50.397943 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:50.397960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:50.399036 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:50.399039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:50.400694 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:50.400709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:50.401519 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:50.401526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:50.401807 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:50.402554 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:50.402576 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:50.404934 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:50.408277 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:50.408287 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:50.408304 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:50.408735 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:50.409110 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:50.409115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:50.409119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:50.409319 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:50.409852 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:50.409860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:50.410214 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 21:34:50.410524 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 21:34:50.411084 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":286,"threads_started":1}
I20260926 21:34:50.413929 17718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42339
I20260926 21:34:50.414258 17718 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:50.415136 17718 heartbeater.cc:511] Master 127.0.97.62:42339 requested a full tablet report, sending...
I20260926 21:34:50.417524 17529 ts_manager.cc:194] Registered new tserver with Master: 2b253f06d1144297b2cf57c315ce3a29 (127.0.97.1:42513)
I20260926 21:34:50.419685 17529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53523
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:34:50.461383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42339
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:50.799863 17724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:50.800148 17724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:50.800241 17724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:50.807775 17724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:50.807963 17724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:50.808142 17724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:34:50.818163 17724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:50.819576 17724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:50.821089 17724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:50.836849 17729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:50.837590 17732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:50.838200 17730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:50.839382 17724 server_base.cc:1034] running on GCE node
I20260926 21:34:50.840229 17724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:50.841538 17724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:50.842782 17724 hybrid_clock.cc:648] HybridClock initialized: now 1790458490842730 us; error 41 us; skew 500 ppm
I20260926 21:34:50.843120 17724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:50 dist-test-slave-87bt krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:50.846380 17724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:34:50.847911 17724 webserver.cc:460] Webserver started at http://127.0.97.2:36583/ using document root <none> and password file <none>
I20260926 21:34:50.848721 17724 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:50.848865 17724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:50.849172 17724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:50.852170 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "68342180fb5a42eb8b6ca1fab0d3eca8"
format_stamp: "Formatted at 2026-09-26 21:34:50 on dist-test-slave-87bt"
server_key: "e281d1010927b195395747acae3a7382"
server_key_iv: "cfb62fc2eb932f75fa61ef32b8d765b9"
server_key_version: "encryptionkey@0"
I20260926 21:34:50.852953 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "68342180fb5a42eb8b6ca1fab0d3eca8"
format_stamp: "Formatted at 2026-09-26 21:34:50 on dist-test-slave-87bt"
server_key: "e281d1010927b195395747acae3a7382"
server_key_iv: "cfb62fc2eb932f75fa61ef32b8d765b9"
server_key_version: "encryptionkey@0"
I20260926 21:34:50.858141 17724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:34:50.861704 17739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:50.863113 17724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:34:50.863312 17724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "68342180fb5a42eb8b6ca1fab0d3eca8"
format_stamp: "Formatted at 2026-09-26 21:34:50 on dist-test-slave-87bt"
server_key: "e281d1010927b195395747acae3a7382"
server_key_iv: "cfb62fc2eb932f75fa61ef32b8d765b9"
server_key_version: "encryptionkey@0"
I20260926 21:34:50.863526 17724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:50.884315 17724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:50.888888 17724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:50.889214 17724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:50.890250 17724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:50.891708 17724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:50.891815 17724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:50.891991 17724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:50.892063 17724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:50.925748 17724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38111
I20260926 21:34:50.925776 17851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38111 every 8 connection(s)
I20260926 21:34:50.927307 17724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:34:50.936286 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 17724
I20260926 21:34:50.936472 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:34:50.936908 388 external_mini_cluster.cc:1428] Setting key c8abfb2b230d9bbf137d6d86841059a8
Sep 26 21:34:50 dist-test-slave-87bt krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:50.948952 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:50.930018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37713 (local address 127.0.97.62:42339)
0926 21:34:50.930539 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:50.930545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:50.931527 (+ 982us) server_negotiation.cc:407] Connection header received
0926 21:34:50.933358 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:50.933362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:50.933432 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:50.933536 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:50.935160 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:50.935920 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:50.937770 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:50.938305 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:50.938411 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:50.942535 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:50.942540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:50.942544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:50.942580 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:50.945149 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:50.946163 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:50.946167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:50.946169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:50.946304 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:50.946845 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:50.946854 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:50.946859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:50.947276 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:50.947441 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:50.948148 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 21:34:50.948418 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260926 21:34:50.949626 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:50.930617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42339 (local address 127.0.97.2:37713)
0926 21:34:50.931342 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:50.931386 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:50.933080 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:50.933771 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:50.933780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:50.934314 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:50.934994 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:50.935012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:50.936073 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:50.936077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:50.937602 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:50.937620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:50.938574 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:50.938581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:50.938915 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:50.939669 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:50.939699 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:50.942209 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:50.945308 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:50.945320 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:50.945342 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:50.946032 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:50.946416 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:50.946440 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:50.946447 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:50.946713 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:50.947410 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:50.947416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:50.947763 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 21:34:50.947998 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 21:34:50.948421 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":270,"threads_started":1}
I20260926 21:34:50.951253 17852 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42339
I20260926 21:34:50.951592 17852 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:50.952344 17852 heartbeater.cc:511] Master 127.0.97.62:42339 requested a full tablet report, sending...
I20260926 21:34:50.953931 17529 ts_manager.cc:194] Registered new tserver with Master: 68342180fb5a42eb8b6ca1fab0d3eca8 (127.0.97.2:38111)
I20260926 21:34:50.954859 17529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37713
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:34:50.993615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42339
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:51.328816 17858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:51.329123 17858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:51.329294 17858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:51.336752 17858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:51.336879 17858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:51.337078 17858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:34:51.347198 17858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:51.348551 17858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:51.349902 17858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:51.363759 17863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:51.363785 17866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:51.363703 17864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:51.364761 17858 server_base.cc:1034] running on GCE node
I20260926 21:34:51.365374 17858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:51.366386 17858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:51.367724 17858 hybrid_clock.cc:648] HybridClock initialized: now 1790458491367647 us; error 49 us; skew 500 ppm
I20260926 21:34:51.368127 17858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:51 dist-test-slave-87bt krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:51.371430 17858 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:34:51.372987 17858 webserver.cc:460] Webserver started at http://127.0.97.3:42543/ using document root <none> and password file <none>
I20260926 21:34:51.373728 17858 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:51.373832 17858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:51.374102 17858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:51.377141 17858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed4425fb1e434d6ca8123e7a3ad5c281"
format_stamp: "Formatted at 2026-09-26 21:34:51 on dist-test-slave-87bt"
server_key: "3df4ba3ca39e2d7d2771c498eb2bea58"
server_key_iv: "2e9c84c7b0c5d2f1b56c32a73c1f8660"
server_key_version: "encryptionkey@0"
I20260926 21:34:51.377907 17858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed4425fb1e434d6ca8123e7a3ad5c281"
format_stamp: "Formatted at 2026-09-26 21:34:51 on dist-test-slave-87bt"
server_key: "3df4ba3ca39e2d7d2771c498eb2bea58"
server_key_iv: "2e9c84c7b0c5d2f1b56c32a73c1f8660"
server_key_version: "encryptionkey@0"
I20260926 21:34:51.383250 17858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 21:34:51.386893 17873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:51.388302 17858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:34:51.388509 17858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "ed4425fb1e434d6ca8123e7a3ad5c281"
format_stamp: "Formatted at 2026-09-26 21:34:51 on dist-test-slave-87bt"
server_key: "3df4ba3ca39e2d7d2771c498eb2bea58"
server_key_iv: "2e9c84c7b0c5d2f1b56c32a73c1f8660"
server_key_version: "encryptionkey@0"
I20260926 21:34:51.388741 17858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:51.423683 17718 heartbeater.cc:503] Master 127.0.97.62:42339 was elected leader, sending a full tablet report...
I20260926 21:34:51.428771 17858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:51.433039 17858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:51.433283 17858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:51.434232 17858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:51.435560 17858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:51.435639 17858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:51.435750 17858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:51.435796 17858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:51.469259 17858 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36595
I20260926 21:34:51.469308 17985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36595 every 8 connection(s)
I20260926 21:34:51.470674 17858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:34:51.478492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 17858
I20260926 21:34:51.478719 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:34:51.479166 388 external_mini_cluster.cc:1428] Setting key 17de901689b407570d5beeb2c101c072
Sep 26 21:34:51 dist-test-slave-87bt krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:51.491285 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.473346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58449 (local address 127.0.97.62:42339)
0926 21:34:51.473812 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.473816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.474772 (+ 956us) server_negotiation.cc:407] Connection header received
0926 21:34:51.476633 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.476641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.476743 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.476863 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:51.478604 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.479513 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.481321 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.481754 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.481894 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.485563 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:51.485568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:51.485571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:51.485632 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:51.488240 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:51.489019 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:51.489023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:51.489025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:51.489148 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:51.489552 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:51.489555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:51.489557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:51.489888 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:51.490003 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.490796 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.491001 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":246,"threads_started":1}
I20260926 21:34:51.492273 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.473866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42339 (local address 127.0.97.3:58449)
0926 21:34:51.474607 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:51.474646 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:51.476356 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:51.477102 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:51.477111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:51.477642 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:51.478372 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.478399 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.479679 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.479684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.481164 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.481177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.481989 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.481998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.482359 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.483109 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:51.483133 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:51.485346 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:51.488370 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:51.488377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:51.488393 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:51.488907 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:51.489270 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:51.489273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:51.489276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:51.489454 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:51.489996 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:51.490002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:51.490364 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 21:34:51.490677 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 21:34:51.491084 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":283,"threads_started":1}
I20260926 21:34:51.493976 17986 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42339
I20260926 21:34:51.494346 17986 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:51.495167 17986 heartbeater.cc:511] Master 127.0.97.62:42339 requested a full tablet report, sending...
I20260926 21:34:51.496592 17529 ts_manager.cc:194] Registered new tserver with Master: ed4425fb1e434d6ca8123e7a3ad5c281 (127.0.97.3:36595)
I20260926 21:34:51.497385 17529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58449
I20260926 21:34:51.507298 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:34:51.524566 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.511885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.0.97.62:42339)
0926 21:34:51.512130 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.512139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.512455 (+ 316us) server_negotiation.cc:407] Connection header received
0926 21:34:51.512588 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.512591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.512656 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.512742 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:51.514100 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.514887 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.516080 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.516481 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.516570 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.519128 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:51.519132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:51.519136 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:51.519168 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:51.521917 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:51.522650 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:51.522654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:51.522656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:51.522771 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:51.523090 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:51.523093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:51.523095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:51.523408 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:51.523586 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.524017 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.524202 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:34:51.528882 17529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:34:51.535775 17529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:34:51.562067 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.548848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38111 (local address 127.0.0.1:59914)
0926 21:34:51.549695 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:51.549744 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:51.549919 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:51.551311 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:51.551323 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:51.551369 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:51.551857 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.551882 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.554512 (+ 2630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.554521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.557765 (+ 3244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.557786 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.560722 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.560732 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.561043 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.561117 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 21:34:51.561295 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 21:34:51.561576 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":998,"threads_started":1}
I20260926 21:34:51.563124 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.549117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.0.97.2:38111)
0926 21:34:51.549621 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.549630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.550155 (+ 525us) server_negotiation.cc:407] Connection header received
0926 21:34:51.550281 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.550291 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.550650 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.550907 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:51.552150 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.554276 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.558012 (+ 3736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.560077 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.560269 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.560547 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.562603 (+ 2056us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.562798 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260926 21:34:51.567673 17787 tablet_service.cc:1467] Processing CreateTablet for tablet cf34a5e612bd459aa91317c8a8888cd3 (DEFAULT_TABLE table=test-table [id=68f5a915568d43f0811f4af491294ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:51.569181 17787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf34a5e612bd459aa91317c8a8888cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:51.570545 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.550772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42513 (local address 127.0.0.1:41842)
0926 21:34:51.562855 (+ 12083us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:51.562873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:51.562976 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:51.564530 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:51.564535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:51.564562 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:51.564749 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.564756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.566091 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.566096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.568029 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.568064 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.569702 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.569707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.569889 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.569921 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:34:51.570023 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 21:34:51.570168 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11971,"mutex_wait_us":129,"thread_start_us":12408,"threads_started":1}
I20260926 21:34:51.570616 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.551360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.0.97.1:42513)
0926 21:34:51.563853 (+ 12493us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.563861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.563889 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:34:51.563982 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.563989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.564202 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.564415 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:51.564890 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.565893 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.568215 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.569526 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.569685 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.569911 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.570086 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.570256 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12289,"thread_start_us":12192,"threads_started":1}
I20260926 21:34:51.571514 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.549906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36595 (local address 127.0.0.1:54176)
0926 21:34:51.562827 (+ 12921us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:51.562854 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:51.563007 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:51.563829 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:51.563835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:51.563869 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:51.564134 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.564141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.566385 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.566390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.568746 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.568764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.570863 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.570869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.571059 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.571093 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:34:51.571178 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 21:34:51.571292 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12741,"thread_start_us":12544,"threads_started":1}
I20260926 21:34:51.572383 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.550388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.0.97.3:36595)
0926 21:34:51.562385 (+ 11997us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.562394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.563160 (+ 766us) server_negotiation.cc:407] Connection header received
0926 21:34:51.563278 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.563286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.563607 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.564297 (+ 690us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:51.564624 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.566206 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.568925 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.570355 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.570505 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.570741 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.571855 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.572030 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11814,"thread_start_us":11638,"threads_started":1}
I20260926 21:34:51.574580 17653 tablet_service.cc:1467] Processing CreateTablet for tablet cf34a5e612bd459aa91317c8a8888cd3 (DEFAULT_TABLE table=test-table [id=68f5a915568d43f0811f4af491294ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:51.575752 17653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf34a5e612bd459aa91317c8a8888cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:51.576061 17921 tablet_service.cc:1467] Processing CreateTablet for tablet cf34a5e612bd459aa91317c8a8888cd3 (DEFAULT_TABLE table=test-table [id=68f5a915568d43f0811f4af491294ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:51.577414 17921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf34a5e612bd459aa91317c8a8888cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:51.584149 18005 tablet_bootstrap.cc:492] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Bootstrap starting.
I20260926 21:34:51.587723 18006 tablet_bootstrap.cc:492] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Bootstrap starting.
I20260926 21:34:51.588390 18005 tablet_bootstrap.cc:654] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:51.589811 18005 log.cc:824] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:51.591529 18006 tablet_bootstrap.cc:654] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:51.592856 18005 tablet_bootstrap.cc:492] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: No bootstrap required, opened a new log
I20260926 21:34:51.592861 18006 log.cc:824] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:51.592916 18008 tablet_bootstrap.cc:492] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Bootstrap starting.
I20260926 21:34:51.593254 18005 ts_tablet_manager.cc:1397] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260926 21:34:51.596534 18008 tablet_bootstrap.cc:654] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:51.597787 18008 log.cc:824] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:51.598829 18006 tablet_bootstrap.cc:492] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: No bootstrap required, opened a new log
I20260926 21:34:51.599174 18006 ts_tablet_manager.cc:1397] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260926 21:34:51.601387 18008 tablet_bootstrap.cc:492] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: No bootstrap required, opened a new log
I20260926 21:34:51.601644 18008 ts_tablet_manager.cc:1397] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260926 21:34:51.603646 18005 raft_consensus.cc:348] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.604115 18005 raft_consensus.cc:374] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:51.604260 18005 raft_consensus.cc:729] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68342180fb5a42eb8b6ca1fab0d3eca8, State: Initialized, Role: FOLLOWER
I20260926 21:34:51.604898 18005 consensus_queue.cc:260] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.606040 17852 heartbeater.cc:503] Master 127.0.97.62:42339 was elected leader, sending a full tablet report...
I20260926 21:34:51.606732 18005 ts_tablet_manager.cc:1428] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260926 21:34:51.607532 18006 raft_consensus.cc:348] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.607985 18006 raft_consensus.cc:374] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:51.608114 18006 raft_consensus.cc:729] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b253f06d1144297b2cf57c315ce3a29, State: Initialized, Role: FOLLOWER
I20260926 21:34:51.608705 18006 consensus_queue.cc:260] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.610520 18006 ts_tablet_manager.cc:1428] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260926 21:34:51.611721 18008 raft_consensus.cc:348] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.612265 18008 raft_consensus.cc:374] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:51.612437 18008 raft_consensus.cc:729] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4425fb1e434d6ca8123e7a3ad5c281, State: Initialized, Role: FOLLOWER
I20260926 21:34:51.613199 18008 consensus_queue.cc:260] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.614259 17986 heartbeater.cc:503] Master 127.0.97.62:42339 was elected leader, sending a full tablet report...
I20260926 21:34:51.614902 18008 ts_tablet_manager.cc:1428] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260926 21:34:51.642884 17719 tablet.cc:2369] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:34:51.679527 17853 tablet.cc:2369] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:34:51.680440 18013 raft_consensus.cc:482] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:34:51.680740 18013 raft_consensus.cc:504] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.682341 18013 leader_election.cc:288] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68342180fb5a42eb8b6ca1fab0d3eca8 (127.0.97.2:38111), 2b253f06d1144297b2cf57c315ce3a29 (127.0.97.1:42513)
I20260926 21:34:51.688933 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.682887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34807 (local address 127.0.97.2:38111)
0926 21:34:51.683086 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.683092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.683115 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:34:51.683214 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.683219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.683293 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.683394 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:51.683932 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.684929 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.686938 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.688084 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.688186 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.688256 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.688597 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.688724 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:34:51.688985 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.682728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38111 (local address 127.0.97.3:34807)
0926 21:34:51.682942 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:51.682963 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:51.683086 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:51.683575 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:51.683579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:51.683604 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:51.683762 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.683769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.685059 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.685065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.686794 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.686805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.688239 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.688245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.688430 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.688467 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 21:34:51.688559 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 21:34:51.688695 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":18}
I20260926 21:34:51.690071 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.683288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58217 (local address 127.0.97.1:42513)
0926 21:34:51.683482 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.683487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.683864 (+ 377us) server_negotiation.cc:407] Connection header received
0926 21:34:51.684013 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.684016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.684100 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.684186 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:51.684740 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.685770 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.687926 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.689042 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.689173 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.689287 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.689751 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.689903 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:34:51.690229 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.683133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42513 (local address 127.0.97.3:58217)
0926 21:34:51.683759 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:51.683772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:51.683891 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:51.684243 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:51.684248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:51.684273 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:51.684590 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.684601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.685939 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.685943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.687770 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:51.687786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.689418 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.689422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:51.689575 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.689615 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 21:34:51.689725 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 21:34:51.689871 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":19,"thread_start_us":338,"threads_started":1}
I20260926 21:34:51.690090 17807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf34a5e612bd459aa91317c8a8888cd3" candidate_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" is_pre_election: true
I20260926 21:34:51.690765 17807 raft_consensus.cc:2454] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4425fb1e434d6ca8123e7a3ad5c281 in term 0.
I20260926 21:34:51.691267 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf34a5e612bd459aa91317c8a8888cd3" candidate_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b253f06d1144297b2cf57c315ce3a29" is_pre_election: true
I20260926 21:34:51.691429 17874 leader_election.cc:302] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68342180fb5a42eb8b6ca1fab0d3eca8, ed4425fb1e434d6ca8123e7a3ad5c281; no voters:
I20260926 21:34:51.691742 17673 raft_consensus.cc:2454] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4425fb1e434d6ca8123e7a3ad5c281 in term 0.
I20260926 21:34:51.691788 18013 raft_consensus.cc:2781] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:34:51.691941 18013 raft_consensus.cc:482] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:34:51.692044 18013 raft_consensus.cc:3037] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:51.694514 18013 raft_consensus.cc:504] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.695169 18013 leader_election.cc:288] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [CANDIDATE]: Term 1 election: Requested vote from peers 68342180fb5a42eb8b6ca1fab0d3eca8 (127.0.97.2:38111), 2b253f06d1144297b2cf57c315ce3a29 (127.0.97.1:42513)
I20260926 21:34:51.695590 17807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf34a5e612bd459aa91317c8a8888cd3" candidate_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8"
I20260926 21:34:51.695716 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf34a5e612bd459aa91317c8a8888cd3" candidate_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b253f06d1144297b2cf57c315ce3a29"
I20260926 21:34:51.695820 17807 raft_consensus.cc:3037] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:51.695978 17673 raft_consensus.cc:3037] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:51.698279 17807 raft_consensus.cc:2454] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4425fb1e434d6ca8123e7a3ad5c281 in term 1.
I20260926 21:34:51.698292 17673 raft_consensus.cc:2454] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4425fb1e434d6ca8123e7a3ad5c281 in term 1.
I20260926 21:34:51.698808 17874 leader_election.cc:302] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68342180fb5a42eb8b6ca1fab0d3eca8, ed4425fb1e434d6ca8123e7a3ad5c281; no voters:
I20260926 21:34:51.699138 18013 raft_consensus.cc:2781] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:34:51.699584 18013 raft_consensus.cc:686] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 1 LEADER]: Becoming Leader. State: Replica: ed4425fb1e434d6ca8123e7a3ad5c281, State: Running, Role: LEADER
I20260926 21:34:51.700057 18013 consensus_queue.cc:237] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } }
I20260926 21:34:51.704772 17529 catalog_manager.cc:5495] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4425fb1e434d6ca8123e7a3ad5c281 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed4425fb1e434d6ca8123e7a3ad5c281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 } health_report { overall_health: UNKNOWN } } }
W20260926 21:34:51.722278 17987 tablet.cc:2369] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:34:51.752512 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.746986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.0.97.3:36595)
0926 21:34:51.747197 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.747203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.747224 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:34:51.747355 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.747361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.747467 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.747612 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:34:51.747949 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.749023 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.750404 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.750830 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.750923 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.751406 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:34:51.751943 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:34:51.752051 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.752141 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.752205 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:34:51.760969 17673 raft_consensus.cc:1261] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 1 FOLLOWER]: Refusing update from remote peer ed4425fb1e434d6ca8123e7a3ad5c281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:34:51.760969 17807 raft_consensus.cc:1261] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 1 FOLLOWER]: Refusing update from remote peer ed4425fb1e434d6ca8123e7a3ad5c281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:34:51.761926 18015 consensus_queue.cc:1035] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68342180fb5a42eb8b6ca1fab0d3eca8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:34:51.762331 18013 consensus_queue.cc:1035] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b253f06d1144297b2cf57c315ce3a29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:34:51.775120 18019 mvcc.cc:205] Tried to move back new op lower bound from 7333717982242136064 to 7333717982007300096. Current Snapshot: MvccSnapshot[applied={T|T < 7333717982242136064}]
I20260926 21:34:51.786639 17528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:34:51.786996 17528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:34:51.791000 17528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fff8b86d8f73430e8be39b5e65d79dec: Sending DeleteTablet for 3 replicas of tablet cf34a5e612bd459aa91317c8a8888cd3
I20260926 21:34:51.792256 17921 tablet_service.cc:1514] Processing DeleteTablet for tablet cf34a5e612bd459aa91317c8a8888cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54176
I20260926 21:34:51.792321 17787 tablet_service.cc:1514] Processing DeleteTablet for tablet cf34a5e612bd459aa91317c8a8888cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59914
I20260926 21:34:51.792570 17653 tablet_service.cc:1514] Processing DeleteTablet for tablet cf34a5e612bd459aa91317c8a8888cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41842
I20260926 21:34:51.795356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 17590
I20260926 21:34:51.805226 18030 tablet_replica.cc:330] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: stopping tablet replica
I20260926 21:34:51.805548 18029 tablet_replica.cc:330] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: stopping tablet replica
I20260926 21:34:51.805786 18030 raft_consensus.cc:2229] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:34:51.806149 18029 raft_consensus.cc:2229] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:34:51.806368 18030 raft_consensus.cc:2258] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:51.806968 18029 raft_consensus.cc:2258] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:51.807802 18028 tablet_replica.cc:330] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: stopping tablet replica
I20260926 21:34:51.808221 18028 raft_consensus.cc:2229] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:34:51.808689 18028 raft_consensus.cc:2258] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:51.810981 18029 ts_tablet_manager.cc:1905] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:51.811650 18030 ts_tablet_manager.cc:1905] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:34:51 dist-test-slave-87bt krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:34:51.818635 18029 ts_tablet_manager.cc:1918] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:51.818635 18030 ts_tablet_manager.cc:1918] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:51.818830 18030 log.cc:1196] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/cf34a5e612bd459aa91317c8a8888cd3
I20260926 21:34:51.818830 18029 log.cc:1196] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/cf34a5e612bd459aa91317c8a8888cd3
I20260926 21:34:51.819391 18029 ts_tablet_manager.cc:1939] T cf34a5e612bd459aa91317c8a8888cd3 P ed4425fb1e434d6ca8123e7a3ad5c281: Deleting consensus metadata
I20260926 21:34:51.819410 18030 ts_tablet_manager.cc:1939] T cf34a5e612bd459aa91317c8a8888cd3 P 2b253f06d1144297b2cf57c315ce3a29: Deleting consensus metadata
I20260926 21:34:51.821391 17516 catalog_manager.cc:4841] TS ed4425fb1e434d6ca8123e7a3ad5c281 (127.0.97.3:36595): tablet cf34a5e612bd459aa91317c8a8888cd3 (table test-table [id=68f5a915568d43f0811f4af491294ef4]) successfully deleted
I20260926 21:34:51.821370 17513 catalog_manager.cc:4841] TS 2b253f06d1144297b2cf57c315ce3a29 (127.0.97.1:42513): tablet cf34a5e612bd459aa91317c8a8888cd3 (table test-table [id=68f5a915568d43f0811f4af491294ef4]) successfully deleted
I20260926 21:34:51.823647 18028 ts_tablet_manager.cc:1905] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:51.825109 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.797131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.0.97.1:42513)
0926 21:34:51.797371 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.797380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.801365 (+ 3985us) server_negotiation.cc:407] Connection header received
0926 21:34:51.802065 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.802074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.802181 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.802271 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:51.806905 (+ 4634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.808003 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.809882 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.810621 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.810770 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.816770 (+ 6000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:51.816787 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:51.816808 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:51.816854 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:51.821367 (+ 4513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:51.822608 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:51.822617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:51.822621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:51.822808 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:51.823299 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:51.823304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:51.823307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:51.823805 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:51.824095 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.824478 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.824729 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 21:34:51.829443 18028 ts_tablet_manager.cc:1918] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:51.829620 18028 log.cc:1196] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/cf34a5e612bd459aa91317c8a8888cd3
I20260926 21:34:51.830196 18028 ts_tablet_manager.cc:1939] T cf34a5e612bd459aa91317c8a8888cd3 P 68342180fb5a42eb8b6ca1fab0d3eca8: Deleting consensus metadata
I20260926 21:34:51.830577 17713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:51.832371 17513 catalog_manager.cc:4841] TS 68342180fb5a42eb8b6ca1fab0d3eca8 (127.0.97.2:38111): tablet cf34a5e612bd459aa91317c8a8888cd3 (table test-table [id=68f5a915568d43f0811f4af491294ef4]) successfully deleted
I20260926 21:34:51.919479 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 17590
I20260926 21:34:51.943030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 17724
Sep 26 21:34:51 dist-test-slave-87bt krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:34:51.960284 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:51.943879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.0.97.2:38111)
0926 21:34:51.944068 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:51.944074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:51.944099 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:34:51.944163 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:51.944169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:51.944263 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:51.944380 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:51.945508 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.946690 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.947989 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:51.948624 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:51.948765 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:51.953128 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:51.953148 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:51.953175 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:51.953226 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:51.956942 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:51.957865 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:51.957876 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:51.957880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:51.958045 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:51.958639 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:51.958643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:51.958645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:51.959034 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:51.959342 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:51.959674 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 21:34:51.959913 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:34:51.961054 17847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:52.050308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 17724
I20260926 21:34:52.068233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 17858
Sep 26 21:34:52 dist-test-slave-87bt krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:34:52.086503 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:52.069100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.0.97.3:36595)
0926 21:34:52.069311 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:52.069317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:52.069340 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:34:52.069476 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:52.069485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:52.069579 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:52.069717 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:52.071321 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:52.072244 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:52.073608 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:52.074068 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:52.074216 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:52.079264 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:52.079284 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:52.079306 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:52.079353 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:52.083443 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:52.084226 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:52.084243 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:52.084247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:52.084404 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:52.084837 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:52.084840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:52.084842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:52.085213 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:52.085561 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:52.085840 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 21:34:52.086068 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:34:52.087231 17981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:52.212563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 17858
I20260926 21:34:52.233103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 17497
I20260926 21:34:52.234373 17559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:52.343271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 17497
2026-09-26T21:34:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4412 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T21:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:34:52Z Disabled control of system clock
I20260926 21:34:52.420221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44703
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:34:52.758416 18039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:52.758734 18039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:52.758904 18039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:52.766360 18039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:34:52.766525 18039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:52.766589 18039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:52.766649 18039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:34:52.766691 18039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:34:52.776129 18039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:52.777375 18039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:52.779139 18039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:52.791186 18044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:52.791282 18045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:52.791150 18047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:52.792254 18039 server_base.cc:1034] running on GCE node
I20260926 21:34:52.793071 18039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:52.794725 18039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:52.795948 18039 hybrid_clock.cc:648] HybridClock initialized: now 1790458492795891 us; error 65 us; skew 500 ppm
I20260926 21:34:52.796316 18039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:34:52.805918 18039 webserver.cc:460] Webserver started at http://127.0.97.62:34861/ using document root <none> and password file <none>
I20260926 21:34:52.806895 18039 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:52.806996 18039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:52.807289 18039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:52.810344 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "69681ccd52aa4765820540a5604d58f2"
format_stamp: "Formatted at 2026-09-26 21:34:52 on dist-test-slave-87bt"
server_key: "c3695a0fab247cdec37236b95f23a34c"
server_key_iv: "89666b079fa7d85cc48055627bf83c19"
server_key_version: "encryptionkey@0"
I20260926 21:34:52.811228 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "69681ccd52aa4765820540a5604d58f2"
format_stamp: "Formatted at 2026-09-26 21:34:52 on dist-test-slave-87bt"
server_key: "c3695a0fab247cdec37236b95f23a34c"
server_key_iv: "89666b079fa7d85cc48055627bf83c19"
server_key_version: "encryptionkey@0"
I20260926 21:34:52.816773 18039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:34:52.820700 18053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:52.822284 18039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:34:52.822523 18039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "69681ccd52aa4765820540a5604d58f2"
format_stamp: "Formatted at 2026-09-26 21:34:52 on dist-test-slave-87bt"
server_key: "c3695a0fab247cdec37236b95f23a34c"
server_key_iv: "89666b079fa7d85cc48055627bf83c19"
server_key_version: "encryptionkey@0"
I20260926 21:34:52.822737 18039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:52.847451 18039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:52.848320 18039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:52.848582 18039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:52.867400 18039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44703
I20260926 21:34:52.867408 18104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44703 every 8 connection(s)
I20260926 21:34:52.868757 18039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:34:52.872547 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:52.875224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 18039
I20260926 21:34:52.875414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:34:52.875963 388 external_mini_cluster.cc:1428] Setting key e9437025810e56f4e9581c9375098966
I20260926 21:34:52.884661 18105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2: Bootstrap starting.
I20260926 21:34:52.888010 18105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:52.889159 18105 log.cc:824] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:52.890074 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:52.878037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45238 (local address 127.0.97.62:44703)
0926 21:34:52.879162 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:52.879173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:52.879216 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:34:52.880864 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:52.880887 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:52.881381 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:52.881800 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:52.882470 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:52.883793 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:52.885246 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:52.886031 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:52.886241 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:52.887449 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:52.887456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:52.887470 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:34:52.888032 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:52.888172 (+ 140us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:34:52.888198 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:52.888324 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:52.888548 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 21:34:52.888847 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":398,"threads_started":1}
I20260926 21:34:52.892385 18105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2: No bootstrap required, opened a new log
I20260926 21:34:52.899775 18105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69681ccd52aa4765820540a5604d58f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44703 } }
I20260926 21:34:52.900135 18105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:52.900213 18105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69681ccd52aa4765820540a5604d58f2, State: Initialized, Role: FOLLOWER
I20260926 21:34:52.900688 18105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69681ccd52aa4765820540a5604d58f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44703 } }
I20260926 21:34:52.900884 18105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:34:52.900990 18105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:34:52.901113 18105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:52.903258 18105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69681ccd52aa4765820540a5604d58f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44703 } }
I20260926 21:34:52.903963 18105 leader_election.cc:302] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69681ccd52aa4765820540a5604d58f2; no voters:
I20260926 21:34:52.904639 18105 leader_election.cc:288] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:34:52.904825 18110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:34:52.905900 18110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [term 1 LEADER]: Becoming Leader. State: Replica: 69681ccd52aa4765820540a5604d58f2, State: Running, Role: LEADER
I20260926 21:34:52.906558 18110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69681ccd52aa4765820540a5604d58f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44703 } }
I20260926 21:34:52.906850 18105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:34:52.909612 18112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69681ccd52aa4765820540a5604d58f2. Latest consensus state: current_term: 1 leader_uuid: "69681ccd52aa4765820540a5604d58f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69681ccd52aa4765820540a5604d58f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44703 } } }
I20260926 21:34:52.910109 18112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:52.910701 18111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69681ccd52aa4765820540a5604d58f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69681ccd52aa4765820540a5604d58f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44703 } } }
I20260926 21:34:52.911088 18111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2 [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:52.912287 18119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:34:52.917963 18119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:34:52.927412 18119 catalog_manager.cc:1347] Generated new cluster ID: 1b628f992abb4351b745c9c9a5c883c6
I20260926 21:34:52.927584 18119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:34:52.952970 18119 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:34:52.954988 18119 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:34:52.963380 18119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2: Generated new TSK 0
I20260926 21:34:52.964476 18119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:34:52.973860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44703
--builtin_ntp_servers=127.0.97.20:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:34:53.310552 18129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:53.310915 18129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:53.311090 18129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:53.318578 18129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:53.318722 18129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:53.318907 18129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:34:53.328620 18129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:53.330137 18129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:53.332027 18129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:53.345265 18134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:53.345360 18135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:53.345382 18137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:53.346299 18129 server_base.cc:1034] running on GCE node
I20260926 21:34:53.346979 18129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:53.347962 18129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:53.349202 18129 hybrid_clock.cc:648] HybridClock initialized: now 1790458493349142 us; error 57 us; skew 500 ppm
I20260926 21:34:53.349529 18129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:34:53.351917 18129 webserver.cc:460] Webserver started at http://127.0.97.1:42307/ using document root <none> and password file <none>
I20260926 21:34:53.352604 18129 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:53.352742 18129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:53.353062 18129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:53.355923 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "c1e61098333f4106b2dd11cd46b67fe0"
format_stamp: "Formatted at 2026-09-26 21:34:53 on dist-test-slave-87bt"
server_key: "fba8679877c868d6e17bf4c7a42a1927"
server_key_iv: "e8cde7ff9a19ff582623aadec70fde11"
server_key_version: "encryptionkey@0"
I20260926 21:34:53.356711 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c1e61098333f4106b2dd11cd46b67fe0"
format_stamp: "Formatted at 2026-09-26 21:34:53 on dist-test-slave-87bt"
server_key: "fba8679877c868d6e17bf4c7a42a1927"
server_key_iv: "e8cde7ff9a19ff582623aadec70fde11"
server_key_version: "encryptionkey@0"
I20260926 21:34:53.361608 18129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:34:53.365108 18143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:53.366532 18129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:34:53.366705 18129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "c1e61098333f4106b2dd11cd46b67fe0"
format_stamp: "Formatted at 2026-09-26 21:34:53 on dist-test-slave-87bt"
server_key: "fba8679877c868d6e17bf4c7a42a1927"
server_key_iv: "e8cde7ff9a19ff582623aadec70fde11"
server_key_version: "encryptionkey@0"
I20260926 21:34:53.366921 18129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:53.395211 18129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:53.396066 18129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:53.396346 18129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:53.397321 18129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:53.398815 18129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:53.398938 18129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:53.399063 18129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:53.399148 18129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:53.432410 18129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44563
I20260926 21:34:53.432502 18255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44563 every 8 connection(s)
I20260926 21:34:53.433745 18129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:34:53.438141 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 18129
I20260926 21:34:53.438330 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:34:53.438793 388 external_mini_cluster.cc:1428] Setting key d1824db25de242fccb51deed8e00330d
I20260926 21:34:53.441394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44703
--builtin_ntp_servers=127.0.97.20:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:34:53.446396 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:53.436263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58417 (local address 127.0.97.62:44703)
0926 21:34:53.436723 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:53.436727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:53.437759 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 21:34:53.439777 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:53.439783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:53.439882 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:53.440001 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:53.441119 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:53.441929 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:53.443539 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:53.443953 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:53.444093 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:53.445531 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:53.445535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:53.445538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:34:53.445558 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:53.445601 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:34:53.445612 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:53.445717 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:53.445952 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 21:34:53.446051 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260926 21:34:53.447480 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:53.436722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44703 (local address 127.0.97.1:58417)
0926 21:34:53.437601 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:53.437635 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:53.439488 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:53.440264 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:53.440274 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:53.440370 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:53.440934 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:53.440955 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:53.442091 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:53.442096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:53.443381 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:53.443393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:53.444199 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:53.444207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:53.444451 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:53.445202 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:34:53.445205 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:53.445252 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:34:53.445336 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:53.445728 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:53.445742 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:53.445756 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 21:34:53.445943 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 21:34:53.446372 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":274,"threads_started":1}
I20260926 21:34:53.449195 18256 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44703
I20260926 21:34:53.449501 18256 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:53.450258 18256 heartbeater.cc:511] Master 127.0.97.62:44703 requested a full tablet report, sending...
I20260926 21:34:53.452649 18070 ts_manager.cc:194] Registered new tserver with Master: c1e61098333f4106b2dd11cd46b67fe0 (127.0.97.1:44563)
I20260926 21:34:53.454880 18070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:58417
W20260926 21:34:53.774782 18260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:53.775060 18260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:53.775142 18260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:53.782356 18260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:53.782477 18260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:53.782588 18260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:34:53.791765 18260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:53.792961 18260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:53.794667 18260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:53.808404 18266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:53.810122 18260 server_base.cc:1034] running on GCE node
W20260926 21:34:53.810529 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:53.810598 18265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:53.811538 18260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:53.812629 18260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:53.813880 18260 hybrid_clock.cc:648] HybridClock initialized: now 1790458493813824 us; error 41 us; skew 500 ppm
I20260926 21:34:53.814293 18260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:34:53.817044 18260 webserver.cc:460] Webserver started at http://127.0.97.2:40527/ using document root <none> and password file <none>
I20260926 21:34:53.817772 18260 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:53.817864 18260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:53.818212 18260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:53.821267 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "1ea29d9dab0f4d56aad72e5d20226224"
format_stamp: "Formatted at 2026-09-26 21:34:53 on dist-test-slave-87bt"
server_key: "31b90e64f7717e646624401415b857f2"
server_key_iv: "0c45d472a90a190bb53ebdd3ea1b1be2"
server_key_version: "encryptionkey@0"
I20260926 21:34:53.822096 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1ea29d9dab0f4d56aad72e5d20226224"
format_stamp: "Formatted at 2026-09-26 21:34:53 on dist-test-slave-87bt"
server_key: "31b90e64f7717e646624401415b857f2"
server_key_iv: "0c45d472a90a190bb53ebdd3ea1b1be2"
server_key_version: "encryptionkey@0"
I20260926 21:34:53.827561 18260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:34:53.831490 18274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:53.833003 18260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:34:53.833194 18260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "1ea29d9dab0f4d56aad72e5d20226224"
format_stamp: "Formatted at 2026-09-26 21:34:53 on dist-test-slave-87bt"
server_key: "31b90e64f7717e646624401415b857f2"
server_key_iv: "0c45d472a90a190bb53ebdd3ea1b1be2"
server_key_version: "encryptionkey@0"
I20260926 21:34:53.833444 18260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:53.853835 18260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:53.854737 18260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:53.855023 18260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:53.856010 18260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:53.857506 18260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:53.857589 18260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:53.857750 18260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:53.857836 18260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:53.915851 18260 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43921
I20260926 21:34:53.915877 18386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43921 every 8 connection(s)
I20260926 21:34:53.917807 18260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:34:53.925767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 18260
I20260926 21:34:53.925969 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:34:53.926599 388 external_mini_cluster.cc:1428] Setting key 1b93244edd5b544e4c0e6a3e3f927dd8
I20260926 21:34:53.929867 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44703
--builtin_ntp_servers=127.0.97.20:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:34:53.939522 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:53.920821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58451 (local address 127.0.97.62:44703)
0926 21:34:53.921037 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:53.921045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:53.922796 (+ 1751us) server_negotiation.cc:407] Connection header received
0926 21:34:53.925960 (+ 3164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:53.925969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:53.926076 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:53.926166 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:53.928112 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:53.929162 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:53.931983 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:53.932762 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:53.932928 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:53.938585 (+ 5657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:53.938593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:53.938600 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:34:53.938633 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:53.938693 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:34:53.938714 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:53.938818 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:53.939114 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 21:34:53.939215 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:34:53.941038 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:53.921392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44703 (local address 127.0.97.2:58451)
0926 21:34:53.922588 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:53.922644 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:53.925536 (+ 2892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:53.926821 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:53.926836 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:53.926982 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:53.927921 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:53.927955 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:53.929378 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:53.929384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:53.931760 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:53.931780 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:53.933014 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:53.933024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:53.933394 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:53.938089 (+ 4695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:34:53.938092 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:53.938164 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:34:53.938269 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:53.938847 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:53.938863 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:53.938875 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 21:34:53.939072 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 21:34:53.939586 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":311,"threads_started":1}
I20260926 21:34:53.942629 18387 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44703
I20260926 21:34:53.942960 18387 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:53.943729 18387 heartbeater.cc:511] Master 127.0.97.62:44703 requested a full tablet report, sending...
I20260926 21:34:53.945184 18070 ts_manager.cc:194] Registered new tserver with Master: 1ea29d9dab0f4d56aad72e5d20226224 (127.0.97.2:43921)
I20260926 21:34:53.945978 18070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:58451
W20260926 21:34:54.265439 18390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:54.265797 18390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:54.265996 18390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:54.273597 18390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:54.273757 18390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:54.273939 18390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:34:54.283253 18390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:54.284605 18390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:54.286394 18390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:54.299808 18395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:54.299872 18398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:54.299841 18396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:54.300792 18390 server_base.cc:1034] running on GCE node
I20260926 21:34:54.301465 18390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:54.302563 18390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:54.303810 18390 hybrid_clock.cc:648] HybridClock initialized: now 1790458494303753 us; error 48 us; skew 500 ppm
I20260926 21:34:54.304177 18390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:34:54.306715 18390 webserver.cc:460] Webserver started at http://127.0.97.3:36229/ using document root <none> and password file <none>
I20260926 21:34:54.307508 18390 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:54.307636 18390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:54.307971 18390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:54.311046 18390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "47fbb16ef91643ddb520b98904da9841"
format_stamp: "Formatted at 2026-09-26 21:34:54 on dist-test-slave-87bt"
server_key: "6d655820133a2a361c882b2bbfcefd3c"
server_key_iv: "5a9dcf3353439ca445fe58aa3dc62886"
server_key_version: "encryptionkey@0"
I20260926 21:34:54.311880 18390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "47fbb16ef91643ddb520b98904da9841"
format_stamp: "Formatted at 2026-09-26 21:34:54 on dist-test-slave-87bt"
server_key: "6d655820133a2a361c882b2bbfcefd3c"
server_key_iv: "5a9dcf3353439ca445fe58aa3dc62886"
server_key_version: "encryptionkey@0"
I20260926 21:34:54.317201 18390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:34:54.320999 18404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:54.322755 18390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 21:34:54.322973 18390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "47fbb16ef91643ddb520b98904da9841"
format_stamp: "Formatted at 2026-09-26 21:34:54 on dist-test-slave-87bt"
server_key: "6d655820133a2a361c882b2bbfcefd3c"
server_key_iv: "5a9dcf3353439ca445fe58aa3dc62886"
server_key_version: "encryptionkey@0"
I20260926 21:34:54.323238 18390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:54.349575 18390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:54.350520 18390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:54.350848 18390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:54.351969 18390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:54.353588 18390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:54.353719 18390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:54.353842 18390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:54.353932 18390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:54.387995 18390 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36659
I20260926 21:34:54.388108 18516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36659 every 8 connection(s)
I20260926 21:34:54.389349 18390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:34:54.398062 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 18390
I20260926 21:34:54.398294 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:34:54.398831 388 external_mini_cluster.cc:1428] Setting key 474f720a3910001c36a2010195e4d716
I20260926 21:34:54.403352 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.392082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49245 (local address 127.0.97.62:44703)
0926 21:34:54.392273 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.392279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.393757 (+ 1478us) server_negotiation.cc:407] Connection header received
0926 21:34:54.395776 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.395782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.395885 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.396003 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:54.397154 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.398059 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.400188 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.400664 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.400800 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.402265 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:54.402271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:54.402274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:34:54.402292 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:54.402325 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:34:54.402337 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:54.402411 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.403005 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.403090 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:34:54.404513 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.392615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44703 (local address 127.0.97.3:49245)
0926 21:34:54.393579 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:54.393621 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:54.395465 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:54.396268 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:54.396277 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:54.396389 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:54.396971 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.396987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.398275 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.398279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.400003 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.400017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.400918 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.400928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.401171 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.401940 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:34:54.401943 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:54.401991 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:34:54.402080 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:54.402532 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:54.402546 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:54.402556 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:34:54.402977 (+ 421us) client_negotiation.cc:240] Negotiation successful
0926 21:34:54.403369 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":305,"threads_started":1}
I20260926 21:34:54.405993 18517 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44703
I20260926 21:34:54.406280 18517 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:54.407223 18517 heartbeater.cc:511] Master 127.0.97.62:44703 requested a full tablet report, sending...
I20260926 21:34:54.408708 18070 ts_manager.cc:194] Registered new tserver with Master: 47fbb16ef91643ddb520b98904da9841 (127.0.97.3:36659)
I20260926 21:34:54.409482 18070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49245
I20260926 21:34:54.413900 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:34:54.424849 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.418414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.0.97.62:44703)
0926 21:34:54.418629 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.418634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.419195 (+ 561us) server_negotiation.cc:407] Connection header received
0926 21:34:54.419351 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.419354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.419431 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.419520 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:54.420184 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.420881 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.422554 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.422992 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.423113 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.424258 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:54.424263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:54.424266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:34:54.424287 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:54.424322 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:34:54.424334 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:54.424408 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.424526 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.424613 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:34:54.429350 18070 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:34:54.436712 18070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:34:54.458173 18256 heartbeater.cc:503] Master 127.0.97.62:44703 was elected leader, sending a full tablet report...
I20260926 21:34:54.471495 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.449396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:44824)
0926 21:34:54.461657 (+ 12261us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:54.461738 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:54.461938 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:54.463459 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:54.463472 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:54.463517 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:54.464065 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.464083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.466233 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.466239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.468804 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.468820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.470564 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.470572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.470725 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.470808 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 21:34:54.470950 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 21:34:54.471148 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12007,"thread_start_us":11833,"threads_started":1}
I20260926 21:34:54.472692 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.450031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.0.97.1:44563)
0926 21:34:54.460776 (+ 10745us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.460790 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.462352 (+ 1562us) server_negotiation.cc:407] Connection header received
0926 21:34:54.462491 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.462499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.462836 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.463101 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:54.464332 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.466050 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.468987 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.470281 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.470417 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.472257 (+ 1840us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.472312 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.472437 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10479,"thread_start_us":10759,"threads_started":1}
I20260926 21:34:54.474820 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.462546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43921 (local address 127.0.0.1:50720)
0926 21:34:54.465662 (+ 3116us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:54.465691 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:54.465933 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:54.467384 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:54.467392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:54.467434 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:54.467977 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.467993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.469578 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.469582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.471725 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.471739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.473802 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.473812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.474180 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.474211 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:34:54.474311 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 21:34:54.474469 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2946,"mutex_wait_us":89,"thread_start_us":2766,"threads_started":1}
I20260926 21:34:54.474939 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.462805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.0.97.2:43921)
0926 21:34:54.465113 (+ 2308us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.465125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.465810 (+ 685us) server_negotiation.cc:407] Connection header received
0926 21:34:54.466156 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.466168 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.466538 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.467002 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:54.468228 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.469436 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.471962 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.473478 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.473618 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.473833 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.474393 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.474588 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2163,"thread_start_us":2067,"threads_started":1}
I20260926 21:34:54.477206 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.449404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36659 (local address 127.0.0.1:42020)
0926 21:34:54.464742 (+ 15338us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:54.464765 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:54.464880 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:54.469007 (+ 4127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:54.469011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:54.469042 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:54.469393 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.469401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.472130 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.472135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.474610 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.474622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.476432 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.476438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.476604 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.476655 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 21:34:54.476751 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 21:34:54.476876 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15195,"mutex_wait_us":82,"thread_start_us":15039,"threads_started":1}
I20260926 21:34:54.477303 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.449759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.0.97.3:36659)
0926 21:34:54.468388 (+ 18629us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.468396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.468432 (+ 36us) server_negotiation.cc:407] Connection header received
0926 21:34:54.468520 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.468527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.468775 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.468970 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:54.469558 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.471992 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.474830 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.476248 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.476402 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.476657 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.476794 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.476937 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18447,"thread_start_us":18293,"threads_started":1}
I20260926 21:34:54.476811 18191 tablet_service.cc:1467] Processing CreateTablet for tablet e285cfa340bf49c388d7aebaff1baaf0 (DEFAULT_TABLE table=test-table [id=a3b197d679554e83b35c8b90bf4a59c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:54.478366 18191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e285cfa340bf49c388d7aebaff1baaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:54.478559 18322 tablet_service.cc:1467] Processing CreateTablet for tablet e285cfa340bf49c388d7aebaff1baaf0 (DEFAULT_TABLE table=test-table [id=a3b197d679554e83b35c8b90bf4a59c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:54.480244 18322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e285cfa340bf49c388d7aebaff1baaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:54.480494 18452 tablet_service.cc:1467] Processing CreateTablet for tablet e285cfa340bf49c388d7aebaff1baaf0 (DEFAULT_TABLE table=test-table [id=a3b197d679554e83b35c8b90bf4a59c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:54.481559 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e285cfa340bf49c388d7aebaff1baaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:54.492363 18535 tablet_bootstrap.cc:492] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Bootstrap starting.
I20260926 21:34:54.495467 18537 tablet_bootstrap.cc:492] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Bootstrap starting.
I20260926 21:34:54.496068 18535 tablet_bootstrap.cc:654] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:54.497157 18536 tablet_bootstrap.cc:492] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Bootstrap starting.
I20260926 21:34:54.497949 18535 log.cc:824] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:54.499121 18537 tablet_bootstrap.cc:654] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:54.500303 18537 log.cc:824] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:54.500509 18536 tablet_bootstrap.cc:654] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:54.501586 18535 tablet_bootstrap.cc:492] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: No bootstrap required, opened a new log
I20260926 21:34:54.501812 18536 log.cc:824] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:54.501869 18535 ts_tablet_manager.cc:1397] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260926 21:34:54.503389 18537 tablet_bootstrap.cc:492] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: No bootstrap required, opened a new log
I20260926 21:34:54.503647 18537 ts_tablet_manager.cc:1397] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260926 21:34:54.504288 18536 tablet_bootstrap.cc:492] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: No bootstrap required, opened a new log
I20260926 21:34:54.504531 18536 ts_tablet_manager.cc:1397] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:34:54.512192 18536 raft_consensus.cc:348] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.512615 18536 raft_consensus.cc:374] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:54.512717 18536 raft_consensus.cc:729] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e61098333f4106b2dd11cd46b67fe0, State: Initialized, Role: FOLLOWER
I20260926 21:34:54.512936 18535 raft_consensus.cc:348] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.513377 18535 raft_consensus.cc:374] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:54.513470 18535 raft_consensus.cc:729] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fbb16ef91643ddb520b98904da9841, State: Initialized, Role: FOLLOWER
I20260926 21:34:54.513398 18536 consensus_queue.cc:260] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.514165 18535 consensus_queue.cc:260] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.514197 18537 raft_consensus.cc:348] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.514616 18537 raft_consensus.cc:374] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:54.514714 18537 raft_consensus.cc:729] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea29d9dab0f4d56aad72e5d20226224, State: Initialized, Role: FOLLOWER
I20260926 21:34:54.515350 18536 ts_tablet_manager.cc:1428] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 21:34:54.515545 18537 consensus_queue.cc:260] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.516485 18517 heartbeater.cc:503] Master 127.0.97.62:44703 was elected leader, sending a full tablet report...
I20260926 21:34:54.517083 18387 heartbeater.cc:503] Master 127.0.97.62:44703 was elected leader, sending a full tablet report...
I20260926 21:34:54.517151 18535 ts_tablet_manager.cc:1428] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 21:34:54.518497 18537 ts_tablet_manager.cc:1428] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260926 21:34:54.641182 18518 tablet.cc:2369] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:34:54.670207 18388 tablet.cc:2369] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:34:54.686298 18257 tablet.cc:2369] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:34:54.708295 18541 raft_consensus.cc:482] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:34:54.708590 18541 raft_consensus.cc:504] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.710270 18541 leader_election.cc:288] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47fbb16ef91643ddb520b98904da9841 (127.0.97.3:36659), 1ea29d9dab0f4d56aad72e5d20226224 (127.0.97.2:43921)
I20260926 21:34:54.717514 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.710703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44435 (local address 127.0.97.2:43921)
0926 21:34:54.710911 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.710917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.711698 (+ 781us) server_negotiation.cc:407] Connection header received
0926 21:34:54.711844 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.711850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.711945 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.712041 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:54.712712 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.713473 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.715748 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.716548 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.716652 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.716751 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.717107 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.717251 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:34:54.717554 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.710683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43921 (local address 127.0.97.1:44435)
0926 21:34:54.711586 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:54.711603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:54.711728 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:54.712065 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:54.712070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:54.712094 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:54.712560 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.712573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.713610 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.713616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.715503 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.715515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.716746 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.716753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.716946 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.716971 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 21:34:54.717055 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 21:34:54.717178 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":196,"thread_start_us":413,"threads_started":1}
I20260926 21:34:54.717659 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.710780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49759 (local address 127.0.97.3:36659)
0926 21:34:54.710988 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.710996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.711851 (+ 855us) server_negotiation.cc:407] Connection header received
0926 21:34:54.712055 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.712061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.712138 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.712239 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:54.712888 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.713611 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.715720 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.716658 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.716783 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.716880 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.717241 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.717372 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:34:54.718134 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.710714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36659 (local address 127.0.97.1:49759)
0926 21:34:54.711723 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:54.711741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:54.711880 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:54.712271 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:54.712275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:54.712297 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:54.712701 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.712712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.713766 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.713770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.715499 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:54.715512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.716865 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.716869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:54.717029 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.717083 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 21:34:54.717189 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 21:34:54.717305 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":55,"thread_start_us":498,"threads_started":1}
I20260926 21:34:54.718772 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e285cfa340bf49c388d7aebaff1baaf0" candidate_uuid: "c1e61098333f4106b2dd11cd46b67fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea29d9dab0f4d56aad72e5d20226224" is_pre_election: true
I20260926 21:34:54.719199 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e285cfa340bf49c388d7aebaff1baaf0" candidate_uuid: "c1e61098333f4106b2dd11cd46b67fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47fbb16ef91643ddb520b98904da9841" is_pre_election: true
I20260926 21:34:54.719415 18342 raft_consensus.cc:2454] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e61098333f4106b2dd11cd46b67fe0 in term 0.
I20260926 21:34:54.719728 18472 raft_consensus.cc:2454] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e61098333f4106b2dd11cd46b67fe0 in term 0.
I20260926 21:34:54.720111 18147 leader_election.cc:302] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea29d9dab0f4d56aad72e5d20226224, c1e61098333f4106b2dd11cd46b67fe0; no voters:
I20260926 21:34:54.720522 18541 raft_consensus.cc:2781] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:34:54.720726 18541 raft_consensus.cc:482] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:34:54.720840 18541 raft_consensus.cc:3037] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:54.723196 18541 raft_consensus.cc:504] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.723903 18541 leader_election.cc:288] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [CANDIDATE]: Term 1 election: Requested vote from peers 47fbb16ef91643ddb520b98904da9841 (127.0.97.3:36659), 1ea29d9dab0f4d56aad72e5d20226224 (127.0.97.2:43921)
I20260926 21:34:54.724428 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e285cfa340bf49c388d7aebaff1baaf0" candidate_uuid: "c1e61098333f4106b2dd11cd46b67fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47fbb16ef91643ddb520b98904da9841"
I20260926 21:34:54.724509 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e285cfa340bf49c388d7aebaff1baaf0" candidate_uuid: "c1e61098333f4106b2dd11cd46b67fe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea29d9dab0f4d56aad72e5d20226224"
I20260926 21:34:54.724653 18472 raft_consensus.cc:3037] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:54.724771 18342 raft_consensus.cc:3037] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:54.727393 18342 raft_consensus.cc:2454] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e61098333f4106b2dd11cd46b67fe0 in term 1.
I20260926 21:34:54.727494 18472 raft_consensus.cc:2454] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e61098333f4106b2dd11cd46b67fe0 in term 1.
I20260926 21:34:54.727962 18147 leader_election.cc:302] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea29d9dab0f4d56aad72e5d20226224, c1e61098333f4106b2dd11cd46b67fe0; no voters:
I20260926 21:34:54.728305 18541 raft_consensus.cc:2781] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:34:54.728820 18541 raft_consensus.cc:686] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 1 LEADER]: Becoming Leader. State: Replica: c1e61098333f4106b2dd11cd46b67fe0, State: Running, Role: LEADER
I20260926 21:34:54.729372 18541 consensus_queue.cc:237] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } }
I20260926 21:34:54.734067 18069 catalog_manager.cc:5495] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1e61098333f4106b2dd11cd46b67fe0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c1e61098333f4106b2dd11cd46b67fe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e61098333f4106b2dd11cd46b67fe0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 } health_report { overall_health: UNKNOWN } } }
I20260926 21:34:54.776501 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.770786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.0.97.1:44563)
0926 21:34:54.770995 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.771003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.771027 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:34:54.771313 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.771317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.771463 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.771660 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:34:54.772026 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.772801 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.774227 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.774685 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.774822 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.775393 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:34:54.775929 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:34:54.776095 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.776153 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.776234 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:34:54.784754 18472 raft_consensus.cc:1261] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 1 FOLLOWER]: Refusing update from remote peer c1e61098333f4106b2dd11cd46b67fe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:34:54.784752 18342 raft_consensus.cc:1261] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 1 FOLLOWER]: Refusing update from remote peer c1e61098333f4106b2dd11cd46b67fe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:34:54.785827 18541 consensus_queue.cc:1035] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47fbb16ef91643ddb520b98904da9841" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:34:54.786119 18546 consensus_queue.cc:1035] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea29d9dab0f4d56aad72e5d20226224" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:34:54.813944 18069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:34:54.814234 18069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:34:54.818218 18069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69681ccd52aa4765820540a5604d58f2: Sending DeleteTablet for 3 replicas of tablet e285cfa340bf49c388d7aebaff1baaf0
I20260926 21:34:54.819451 18191 tablet_service.cc:1514] Processing DeleteTablet for tablet e285cfa340bf49c388d7aebaff1baaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:54 UTC) from {username='slave'} at 127.0.0.1:44824
I20260926 21:34:54.820258 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet e285cfa340bf49c388d7aebaff1baaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:54 UTC) from {username='slave'} at 127.0.0.1:42020
I20260926 21:34:54.820508 18559 tablet_replica.cc:330] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: stopping tablet replica
I20260926 21:34:54.820976 18559 raft_consensus.cc:2229] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:34:54.821075 18560 tablet_replica.cc:330] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: stopping tablet replica
I20260926 21:34:54.821602 18560 raft_consensus.cc:2229] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:34:54.821705 18559 raft_consensus.cc:2258] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:54.821709 18322 tablet_service.cc:1514] Processing DeleteTablet for tablet e285cfa340bf49c388d7aebaff1baaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:54 UTC) from {username='slave'} at 127.0.0.1:50720
I20260926 21:34:54.822259 18560 raft_consensus.cc:2258] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:54.824378 18560 ts_tablet_manager.cc:1905] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:54.824750 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 18129
I20260926 21:34:54.833992 18559 ts_tablet_manager.cc:1905] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:54.837796 18560 ts_tablet_manager.cc:1918] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:54.838143 18560 log.cc:1196] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/e285cfa340bf49c388d7aebaff1baaf0
I20260926 21:34:54.838802 18560 ts_tablet_manager.cc:1939] T e285cfa340bf49c388d7aebaff1baaf0 P 47fbb16ef91643ddb520b98904da9841: Deleting consensus metadata
I20260926 21:34:54.840888 18054 catalog_manager.cc:4841] TS 47fbb16ef91643ddb520b98904da9841 (127.0.97.3:36659): tablet e285cfa340bf49c388d7aebaff1baaf0 (table test-table [id=a3b197d679554e83b35c8b90bf4a59c5]) successfully deleted
I20260926 21:34:54.843696 18561 tablet_replica.cc:330] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: stopping tablet replica
I20260926 21:34:54.844172 18561 raft_consensus.cc:2229] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:34:54.844717 18561 raft_consensus.cc:2258] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:54.845117 18559 ts_tablet_manager.cc:1918] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:54.845295 18559 log.cc:1196] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/e285cfa340bf49c388d7aebaff1baaf0
I20260926 21:34:54.845719 18559 ts_tablet_manager.cc:1939] T e285cfa340bf49c388d7aebaff1baaf0 P c1e61098333f4106b2dd11cd46b67fe0: Deleting consensus metadata
I20260926 21:34:54.847431 18057 catalog_manager.cc:4841] TS c1e61098333f4106b2dd11cd46b67fe0 (127.0.97.1:44563): tablet e285cfa340bf49c388d7aebaff1baaf0 (table test-table [id=a3b197d679554e83b35c8b90bf4a59c5]) successfully deleted
I20260926 21:34:54.849702 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.826525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.0.97.1:44563)
0926 21:34:54.826781 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.826793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.833695 (+ 6902us) server_negotiation.cc:407] Connection header received
0926 21:34:54.835295 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.835301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.835431 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.835509 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:54.841730 (+ 6221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.843054 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.845504 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.846207 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.846318 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.848042 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:54.848055 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:54.848078 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:34:54.848886 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:54.848946 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:34:54.848971 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:54.849083 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.849226 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.849334 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 21:34:54.850414 18251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:54.851464 18561 ts_tablet_manager.cc:1905] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:54.857744 18561 ts_tablet_manager.cc:1918] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:54.857897 18561 log.cc:1196] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/e285cfa340bf49c388d7aebaff1baaf0
I20260926 21:34:54.858376 18561 ts_tablet_manager.cc:1939] T e285cfa340bf49c388d7aebaff1baaf0 P 1ea29d9dab0f4d56aad72e5d20226224: Deleting consensus metadata
I20260926 21:34:54.860042 18057 catalog_manager.cc:4841] TS 1ea29d9dab0f4d56aad72e5d20226224 (127.0.97.2:43921): tablet e285cfa340bf49c388d7aebaff1baaf0 (table test-table [id=a3b197d679554e83b35c8b90bf4a59c5]) successfully deleted
I20260926 21:34:54.948164 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 18129
I20260926 21:34:54.962598 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 18260
I20260926 21:34:54.969666 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:54.963368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.0.97.2:43921)
0926 21:34:54.963546 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:54.963552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:54.963650 (+ 98us) server_negotiation.cc:407] Connection header received
0926 21:34:54.963888 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:54.963893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:54.963969 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:54.964079 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:54.964487 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.965241 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.966726 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:54.967116 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:54.967259 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:54.968353 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:54.968362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:54.968376 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:34:54.968955 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:54.968999 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:34:54.969017 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:54.969137 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:54.969241 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 21:34:54.969348 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:34:54.970326 18382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:55.059401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 18260
I20260926 21:34:55.076488 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 18390
I20260926 21:34:55.082695 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:55.077217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42024 (local address 127.0.97.3:36659)
0926 21:34:55.077377 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:55.077381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:55.077399 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:34:55.077490 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:55.077494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:55.077572 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:55.077671 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:55.078042 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:55.078816 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:55.080037 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:55.080423 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:55.080511 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:55.081447 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:55.081456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:55.081471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:34:55.082041 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:55.082088 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:34:55.082106 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:55.082205 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:55.082299 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 21:34:55.082380 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:34:55.083338 18512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:55.170342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 18390
I20260926 21:34:55.186578 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 18039
I20260926 21:34:55.187629 18100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:55.295387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 18039
2026-09-26T21:34:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2949 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:34:55 dist-test-slave-87bt krb5kdc[18569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:34:55 dist-test-slave-87bt krb5kdc[18569](info): set up 2 sockets
Sep 26 21:34:55 dist-test-slave-87bt krb5kdc[18569](info): commencing operation
krb5kdc: starting...
W20260926 21:34:56.398561 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:34:56 dist-test-slave-87bt krb5kdc[18569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:34:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:34:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:34:56.649886 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40047
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:56.998317 18582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:56.998682 18582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:56.998903 18582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:57.006453 18582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:34:57.006575 18582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:57.006697 18582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:57.006817 18582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:34:57.006870 18582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:34:57.017549 18582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:57.019155 18582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:57.020592 18582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:57.034379 18587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:57.034629 18582 server_base.cc:1034] running on GCE node
W20260926 21:34:57.034312 18590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:57.034456 18588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:57.036132 18582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:57.037779 18582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:57.039064 18582 hybrid_clock.cc:648] HybridClock initialized: now 1790458497038996 us; error 58 us; skew 500 ppm
I20260926 21:34:57.039539 18582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:57 dist-test-slave-87bt krb5kdc[18569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:57.043025 18582 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:34:57.044808 18582 webserver.cc:460] Webserver started at http://127.0.97.62:44517/ using document root <none> and password file <none>
I20260926 21:34:57.045640 18582 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:57.045800 18582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:57.046137 18582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:57.049332 18582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "83df1682407746a8a8307e510895faf4"
format_stamp: "Formatted at 2026-09-26 21:34:57 on dist-test-slave-87bt"
server_key: "44ede955591aa67d19a693a8ba522a42"
server_key_iv: "ab1f206a54cf3ea5c8347e61ecc6f0fd"
server_key_version: "encryptionkey@0"
I20260926 21:34:57.050158 18582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "83df1682407746a8a8307e510895faf4"
format_stamp: "Formatted at 2026-09-26 21:34:57 on dist-test-slave-87bt"
server_key: "44ede955591aa67d19a693a8ba522a42"
server_key_iv: "ab1f206a54cf3ea5c8347e61ecc6f0fd"
server_key_version: "encryptionkey@0"
I20260926 21:34:57.055788 18582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:34:57.059649 18597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:57.061034 18582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:34:57.061239 18582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "83df1682407746a8a8307e510895faf4"
format_stamp: "Formatted at 2026-09-26 21:34:57 on dist-test-slave-87bt"
server_key: "44ede955591aa67d19a693a8ba522a42"
server_key_iv: "ab1f206a54cf3ea5c8347e61ecc6f0fd"
server_key_version: "encryptionkey@0"
I20260926 21:34:57.061455 18582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:57.075160 18582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:57.079725 18582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:57.080049 18582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:57.099949 18582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40047
I20260926 21:34:57.099973 18648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40047 every 8 connection(s)
I20260926 21:34:57.101488 18582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:34:57.104884 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 18582
I20260926 21:34:57.105069 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:34:57.105587 388 external_mini_cluster.cc:1428] Setting key 6ec7c37f73308c57338cb98290780068
I20260926 21:34:57.105983 18649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:57.121472 18649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4: Bootstrap starting.
I20260926 21:34:57.124917 18649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:57.126000 18649 log.cc:824] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:34:57 dist-test-slave-87bt krb5kdc[18569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:57.129238 18649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4: No bootstrap required, opened a new log
I20260926 21:34:57.135736 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:57.107578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.0.97.62:40047)
0926 21:34:57.115017 (+ 7439us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:57.115034 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:57.115082 (+ 48us) server_negotiation.cc:407] Connection header received
0926 21:34:57.116915 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:57.116942 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:57.117385 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:57.117818 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:57.119465 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:57.120806 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:57.122270 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:57.123068 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:57.123276 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:57.127891 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:57.127899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:57.127915 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:57.127953 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:57.131189 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:57.132187 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:57.132193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:57.132201 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:57.132375 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:57.132816 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:57.132820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:57.132822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:57.133382 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:57.133589 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:57.134096 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 21:34:57.134538 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7189,"thread_start_us":6965,"threads_started":1}
I20260926 21:34:57.138520 18649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83df1682407746a8a8307e510895faf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260926 21:34:57.138978 18649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:57.139079 18649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83df1682407746a8a8307e510895faf4, State: Initialized, Role: FOLLOWER
I20260926 21:34:57.139590 18649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83df1682407746a8a8307e510895faf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260926 21:34:57.139827 18649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:34:57.139925 18649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:34:57.140048 18649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:57.142513 18649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83df1682407746a8a8307e510895faf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260926 21:34:57.143215 18649 leader_election.cc:302] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83df1682407746a8a8307e510895faf4; no voters:
I20260926 21:34:57.143875 18649 leader_election.cc:288] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:34:57.144011 18654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:34:57.144764 18654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [term 1 LEADER]: Becoming Leader. State: Replica: 83df1682407746a8a8307e510895faf4, State: Running, Role: LEADER
I20260926 21:34:57.145399 18654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83df1682407746a8a8307e510895faf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260926 21:34:57.146317 18649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:34:57.149084 18655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83df1682407746a8a8307e510895faf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83df1682407746a8a8307e510895faf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } } }
I20260926 21:34:57.149613 18655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:57.149084 18656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83df1682407746a8a8307e510895faf4. Latest consensus state: current_term: 1 leader_uuid: "83df1682407746a8a8307e510895faf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83df1682407746a8a8307e510895faf4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } } }
I20260926 21:34:57.149748 18656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4 [sys.catalog]: This master's current role is: LEADER
I20260926 21:34:57.151625 18660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:34:57.157435 18660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:34:57.167330 18660 catalog_manager.cc:1347] Generated new cluster ID: 96026dd37127418ab5a2afa09be4840d
I20260926 21:34:57.167459 18660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:34:57.193145 18660 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:34:57.194752 18660 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:34:57.200795 18660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4: Generated new TSK 0
I20260926 21:34:57.201716 18660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:34:57.273897 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:57.644915 18675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:57.645334 18675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:57.645511 18675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:57.653455 18675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:57.653627 18675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:57.653796 18675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:34:57.663723 18675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:57.665076 18675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:57.666635 18675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:57.679520 18680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:57.679493 18681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:57.679852 18683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:57.680696 18675 server_base.cc:1034] running on GCE node
I20260926 21:34:57.681318 18675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:57.682341 18675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:57.683537 18675 hybrid_clock.cc:648] HybridClock initialized: now 1790458497683497 us; error 37 us; skew 500 ppm
I20260926 21:34:57.683864 18675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:57 dist-test-slave-87bt krb5kdc[18569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:57.687004 18675 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:34:57.688331 18675 webserver.cc:460] Webserver started at http://127.0.97.1:34437/ using document root <none> and password file <none>
I20260926 21:34:57.688953 18675 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:57.689049 18675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:57.689301 18675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:57.692272 18675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "6b9550880bb94d769dbc8bd0ba93e91e"
format_stamp: "Formatted at 2026-09-26 21:34:57 on dist-test-slave-87bt"
server_key: "87680b37b0fb27c9f8bbe052c39974c3"
server_key_iv: "827e5ceb8b80a54e306c36f80400ca4a"
server_key_version: "encryptionkey@0"
I20260926 21:34:57.693003 18675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6b9550880bb94d769dbc8bd0ba93e91e"
format_stamp: "Formatted at 2026-09-26 21:34:57 on dist-test-slave-87bt"
server_key: "87680b37b0fb27c9f8bbe052c39974c3"
server_key_iv: "827e5ceb8b80a54e306c36f80400ca4a"
server_key_version: "encryptionkey@0"
I20260926 21:34:57.697927 18675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:34:57.701356 18690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:57.702740 18675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:34:57.702920 18675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "6b9550880bb94d769dbc8bd0ba93e91e"
format_stamp: "Formatted at 2026-09-26 21:34:57 on dist-test-slave-87bt"
server_key: "87680b37b0fb27c9f8bbe052c39974c3"
server_key_iv: "827e5ceb8b80a54e306c36f80400ca4a"
server_key_version: "encryptionkey@0"
I20260926 21:34:57.703132 18675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:57.715099 18675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:57.719327 18675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:57.719621 18675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:57.720533 18675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:57.721837 18675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:57.721940 18675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:57.722086 18675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:57.722164 18675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:57.756163 18675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44561
I20260926 21:34:57.756204 18802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44561 every 8 connection(s)
I20260926 21:34:57.757519 18675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:34:57.763527 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 18675
I20260926 21:34:57.763711 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:34:57.764331 388 external_mini_cluster.cc:1428] Setting key ad42211d9ad10de3d291ca78e9b35ee9
Sep 26 21:34:57 dist-test-slave-87bt krb5kdc[18569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:57.782595 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:57.760357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57197 (local address 127.0.97.62:40047)
0926 21:34:57.760820 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:57.760826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:57.761892 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 21:34:57.764450 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:57.764456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:57.764551 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:57.764654 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:57.766819 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:57.767500 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:57.770302 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:57.770751 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:57.770844 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:57.775725 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:57.775730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:57.775733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:57.775768 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:57.779263 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:57.780005 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:57.780013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:57.780019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:57.780188 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:57.780587 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:57.780594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:57.780600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:57.780982 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:57.781143 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:57.781820 (+ 677us) server_negotiation.cc:299] Negotiation successful
0926 21:34:57.782075 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":225,"threads_started":1}
I20260926 21:34:57.783262 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:57.760960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40047 (local address 127.0.97.1:57197)
0926 21:34:57.761717 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:57.761762 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:57.764085 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:57.765001 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:57.765022 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:57.765784 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:57.766638 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:57.766665 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:57.767667 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:57.767676 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:57.770140 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:57.770163 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:57.770998 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:57.771008 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:57.771385 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:57.772154 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:57.772191 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:57.775467 (+ 3276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:57.779405 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:57.779412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:57.779426 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:57.779863 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:57.780293 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:57.780296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:57.780298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:57.780469 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:57.781097 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:57.781104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:57.781435 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 21:34:57.781662 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 21:34:57.782051 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":313,"threads_started":1}
I20260926 21:34:57.784919 18803 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40047
I20260926 21:34:57.785249 18803 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:57.785965 18803 heartbeater.cc:511] Master 127.0.97.62:40047 requested a full tablet report, sending...
I20260926 21:34:57.788321 18614 ts_manager.cc:194] Registered new tserver with Master: 6b9550880bb94d769dbc8bd0ba93e91e (127.0.97.1:44561)
I20260926 21:34:57.790704 18614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57197
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:34:57.823645 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:58.152539 18809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:58.152810 18809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:58.152904 18809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:58.160115 18809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:58.160216 18809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:58.160324 18809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:34:58.170126 18809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:58.171455 18809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:58.172740 18809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:58.185941 18814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:58.185977 18815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:58.186208 18817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:58.186887 18809 server_base.cc:1034] running on GCE node
I20260926 21:34:58.187434 18809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:58.188345 18809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:58.189548 18809 hybrid_clock.cc:648] HybridClock initialized: now 1790458498189494 us; error 40 us; skew 500 ppm
I20260926 21:34:58.189877 18809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:58 dist-test-slave-87bt krb5kdc[18569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:58.197858 18809 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:34:58.199474 18809 webserver.cc:460] Webserver started at http://127.0.97.2:45593/ using document root <none> and password file <none>
I20260926 21:34:58.200208 18809 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:58.200356 18809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:58.200682 18809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:58.203842 18809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "7c95e230854a400f869a1c9ffaaee067"
format_stamp: "Formatted at 2026-09-26 21:34:58 on dist-test-slave-87bt"
server_key: "24bae46cb528f88a441fd688a1f6fa84"
server_key_iv: "ef302fcdcc6922aadac239fd6c889fb8"
server_key_version: "encryptionkey@0"
I20260926 21:34:58.204698 18809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7c95e230854a400f869a1c9ffaaee067"
format_stamp: "Formatted at 2026-09-26 21:34:58 on dist-test-slave-87bt"
server_key: "24bae46cb528f88a441fd688a1f6fa84"
server_key_iv: "ef302fcdcc6922aadac239fd6c889fb8"
server_key_version: "encryptionkey@0"
I20260926 21:34:58.210125 18809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:34:58.213742 18824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:58.215166 18809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 21:34:58.215351 18809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "7c95e230854a400f869a1c9ffaaee067"
format_stamp: "Formatted at 2026-09-26 21:34:58 on dist-test-slave-87bt"
server_key: "24bae46cb528f88a441fd688a1f6fa84"
server_key_iv: "ef302fcdcc6922aadac239fd6c889fb8"
server_key_version: "encryptionkey@0"
I20260926 21:34:58.215556 18809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:58.237370 18809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:58.242128 18809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:58.242470 18809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:58.243618 18809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:58.245005 18809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:58.245114 18809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:58.245261 18809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:58.245338 18809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:58.279287 18809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41131
I20260926 21:34:58.279337 18936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41131 every 8 connection(s)
I20260926 21:34:58.280575 18809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:34:58.288295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 18809
I20260926 21:34:58.288487 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:34:58.288923 388 external_mini_cluster.cc:1428] Setting key 0e90ce469f02d2a06e35fca28bdcd0ae
Sep 26 21:34:58 dist-test-slave-87bt krb5kdc[18569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:58.300931 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.283270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33967 (local address 127.0.97.62:40047)
0926 21:34:58.283679 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:58.283684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:58.284735 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 21:34:58.286519 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:58.286523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:58.286596 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:58.286681 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:58.288103 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.288948 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.291153 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.291591 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.291730 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.295207 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:58.295212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:58.295216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:58.295247 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:58.297850 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:58.298615 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:58.298623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:58.298629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:58.298766 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:58.299194 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:58.299198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:58.299201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:58.299462 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:58.299618 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:58.300323 (+ 705us) server_negotiation.cc:299] Negotiation successful
0926 21:34:58.300540 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":210,"threads_started":1}
I20260926 21:34:58.301882 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.283875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40047 (local address 127.0.97.2:33967)
0926 21:34:58.284549 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:58.284587 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:58.286232 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:58.286861 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:58.286868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:58.287356 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:58.287906 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.287923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.289145 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.289149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.290988 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.291003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.291829 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.291836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.292062 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.292754 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:58.292782 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:58.294966 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:58.297998 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:58.298008 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:58.298025 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:58.298497 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:58.298880 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:58.298885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:58.298889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:58.299077 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:58.299590 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:58.299599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:58.299944 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 21:34:58.300212 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 21:34:58.300787 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":265,"threads_started":1}
I20260926 21:34:58.303330 18937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40047
I20260926 21:34:58.303664 18937 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:58.304517 18937 heartbeater.cc:511] Master 127.0.97.62:40047 requested a full tablet report, sending...
I20260926 21:34:58.306049 18614 ts_manager.cc:194] Registered new tserver with Master: 7c95e230854a400f869a1c9ffaaee067 (127.0.97.2:41131)
I20260926 21:34:58.306963 18614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33967
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:34:58.343302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:34:58.675335 18943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:34:58.675668 18943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:34:58.675843 18943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:34:58.683097 18943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:34:58.683256 18943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:34:58.683441 18943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:34:58.693293 18943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:34:58.694715 18943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:34:58.696033 18943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:34:58.709357 18948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:58.709316 18951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:34:58.709381 18949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:34:58.710119 18943 server_base.cc:1034] running on GCE node
I20260926 21:34:58.710861 18943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:34:58.711746 18943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:34:58.712996 18943 hybrid_clock.cc:648] HybridClock initialized: now 1790458498712935 us; error 60 us; skew 500 ppm
I20260926 21:34:58.713325 18943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:34:58 dist-test-slave-87bt krb5kdc[18569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:34:58.716377 18943 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:34:58.717747 18943 webserver.cc:460] Webserver started at http://127.0.97.3:42755/ using document root <none> and password file <none>
I20260926 21:34:58.718482 18943 fs_manager.cc:359] Metadata directory not provided
I20260926 21:34:58.718647 18943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:34:58.718976 18943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:34:58.722025 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "87fca72c5f2244b4b90cb754ccf889ee"
format_stamp: "Formatted at 2026-09-26 21:34:58 on dist-test-slave-87bt"
server_key: "d11e4cd9092004fe6ca6e0b25454de65"
server_key_iv: "df00da5564156236a9dbac0ca4d64dfc"
server_key_version: "encryptionkey@0"
I20260926 21:34:58.722857 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "87fca72c5f2244b4b90cb754ccf889ee"
format_stamp: "Formatted at 2026-09-26 21:34:58 on dist-test-slave-87bt"
server_key: "d11e4cd9092004fe6ca6e0b25454de65"
server_key_iv: "df00da5564156236a9dbac0ca4d64dfc"
server_key_version: "encryptionkey@0"
I20260926 21:34:58.728116 18943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:34:58.732079 18958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:58.733532 18943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:34:58.733737 18943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "87fca72c5f2244b4b90cb754ccf889ee"
format_stamp: "Formatted at 2026-09-26 21:34:58 on dist-test-slave-87bt"
server_key: "d11e4cd9092004fe6ca6e0b25454de65"
server_key_iv: "df00da5564156236a9dbac0ca4d64dfc"
server_key_version: "encryptionkey@0"
I20260926 21:34:58.733960 18943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:34:58.758636 18943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:34:58.763257 18943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:34:58.763499 18943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:34:58.764451 18943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:34:58.765818 18943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:34:58.765936 18943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:58.766075 18943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:34:58.766139 18943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:34:58.794462 18803 heartbeater.cc:503] Master 127.0.97.62:40047 was elected leader, sending a full tablet report...
I20260926 21:34:58.797485 18943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41299
I20260926 21:34:58.797569 19070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41299 every 8 connection(s)
I20260926 21:34:58.798807 18943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:34:58.801741 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 18943
I20260926 21:34:58.801914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:34:58.802523 388 external_mini_cluster.cc:1428] Setting key fb3466f3230a2ed4468cca987e7ef44f
Sep 26 21:34:58 dist-test-slave-87bt krb5kdc[18569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:34:58.819092 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.801444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55419 (local address 127.0.97.62:40047)
0926 21:34:58.801941 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:58.801945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:58.805243 (+ 3298us) server_negotiation.cc:407] Connection header received
0926 21:34:58.805291 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:58.805295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:58.805365 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:58.805446 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:58.807124 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.807946 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.809436 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.809823 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.809913 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.813257 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:58.813262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:58.813266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:58.813296 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:58.815776 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:58.816708 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:58.816713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:58.816717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:58.816864 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:58.817372 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:58.817375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:58.817377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:58.817763 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:58.817890 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:58.818646 (+ 756us) server_negotiation.cc:299] Negotiation successful
0926 21:34:58.818822 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":250,"threads_started":1}
I20260926 21:34:58.820215 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.802004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40047 (local address 127.0.97.3:55419)
0926 21:34:58.802878 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:58.802915 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:58.804901 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:58.805676 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:58.805685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:58.806285 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:34:58.806940 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.806959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.808071 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.808074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.809311 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.809323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.810005 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.810010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.810216 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.810949 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:34:58.810972 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:34:58.813056 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:34:58.815921 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:58.815928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:58.815941 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:58.816576 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:58.816986 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:34:58.816991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:34:58.816996 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:34:58.817258 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:34:58.817905 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:34:58.817911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:34:58.818249 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 21:34:58.818537 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 21:34:58.818996 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":387,"threads_started":1}
I20260926 21:34:58.821713 19071 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40047
I20260926 21:34:58.822034 19071 heartbeater.cc:464] Registering TS with master...
I20260926 21:34:58.822816 19071 heartbeater.cc:511] Master 127.0.97.62:40047 requested a full tablet report, sending...
I20260926 21:34:58.824280 18614 ts_manager.cc:194] Registered new tserver with Master: 87fca72c5f2244b4b90cb754ccf889ee (127.0.97.3:41299)
I20260926 21:34:58.825095 18614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55419
I20260926 21:34:58.830536 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:34:58.851399 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.837989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.97.62:40047)
0926 21:34:58.838194 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:58.838199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:58.838744 (+ 545us) server_negotiation.cc:407] Connection header received
0926 21:34:58.838884 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:58.838888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:58.838977 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:58.839077 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:58.840512 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.841269 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.842627 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.843114 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.843214 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.845792 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:58.845797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:58.845800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:58.845829 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:58.848674 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:58.849469 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:58.849473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:58.849476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:58.849601 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:58.850011 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:58.850015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:58.850017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:58.850332 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:58.850588 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:58.850934 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 21:34:58.851077 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:34:58.855175 18614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:34:58.861955 18614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:34:58.902624 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.874767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.0.97.3:41299)
0926 21:34:58.895253 (+ 20486us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:58.895260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:58.895284 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:34:58.895351 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:58.895356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:58.895507 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:58.895703 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:58.896245 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.897564 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.900000 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.901383 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.901507 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.901727 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:58.902012 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 21:34:58.902193 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20336,"thread_start_us":20177,"threads_started":1}
I20260926 21:34:58.902763 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.877085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.0.97.1:44561)
0926 21:34:58.892462 (+ 15377us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:58.892475 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:58.893580 (+ 1105us) server_negotiation.cc:407] Connection header received
0926 21:34:58.893936 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:58.893948 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:58.894256 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:58.894502 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:58.895170 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.896953 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.899909 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.901221 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.901342 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.901578 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:58.902276 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 21:34:58.902470 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15182,"thread_start_us":15030,"threads_started":1}
I20260926 21:34:58.902349 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.874515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41131 (local address 127.0.0.1:35462)
0926 21:34:58.891677 (+ 17162us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:58.891701 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:58.891859 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:58.893694 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:58.893702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:58.893730 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:58.894184 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.894197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.896450 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.896455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.899131 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.899145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.901249 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.901255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.901464 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.901525 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 21:34:58.901676 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 21:34:58.901863 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17021,"mutex_wait_us":29,"thread_start_us":18569,"threads_started":1}
I20260926 21:34:58.902513 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.874309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41299 (local address 127.0.0.1:37936)
0926 21:34:58.890901 (+ 16592us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:58.890962 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:58.891256 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:58.895690 (+ 4434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:58.895701 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:58.895727 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:58.896070 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.896079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.897736 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.897741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.899816 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.899831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.901614 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.901622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.901819 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.901855 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 21:34:58.901950 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 21:34:58.902089 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16207,"mutex_wait_us":150,"thread_start_us":16020,"threads_started":1}
I20260926 21:34:58.903594 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.876448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44636)
0926 21:34:58.893427 (+ 16979us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:58.893450 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:58.893701 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:58.894541 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:58.894546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:58.894574 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:58.894967 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.894978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.897560 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.897565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.899754 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.899764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.901709 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.901714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.901874 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.901915 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 21:34:58.901979 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 21:34:58.902108 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16816,"mutex_wait_us":68,"thread_start_us":14926,"threads_started":1}
I20260926 21:34:58.903790 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.874760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.0.97.2:41131)
0926 21:34:58.892962 (+ 18202us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:58.892974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:58.893012 (+ 38us) server_negotiation.cc:407] Connection header received
0926 21:34:58.893154 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:58.893165 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:58.893443 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:58.894399 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:58.894682 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.896300 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.899292 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.900798 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.900927 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.901129 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:58.903267 (+ 2138us) server_negotiation.cc:299] Negotiation successful
0926 21:34:58.903446 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17998,"thread_start_us":17892,"threads_started":1}
I20260926 21:34:58.907047 18872 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb2719448ce42949ce68f4dfe419bed (DEFAULT_TABLE table=test-table [id=55266fc2bbf74160b1f7d71e28283914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:58.907050 19006 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb2719448ce42949ce68f4dfe419bed (DEFAULT_TABLE table=test-table [id=55266fc2bbf74160b1f7d71e28283914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:58.908264 18872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb2719448ce42949ce68f4dfe419bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:58.908264 19006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb2719448ce42949ce68f4dfe419bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:58.917048 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb2719448ce42949ce68f4dfe419bed (DEFAULT_TABLE table=test-table [id=55266fc2bbf74160b1f7d71e28283914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:34:58.918525 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb2719448ce42949ce68f4dfe419bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:34:58.919341 19090 tablet_bootstrap.cc:492] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Bootstrap starting.
I20260926 21:34:58.922097 19091 tablet_bootstrap.cc:492] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Bootstrap starting.
I20260926 21:34:58.922869 19090 tablet_bootstrap.cc:654] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:58.924513 19090 log.cc:824] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:58.925657 19091 tablet_bootstrap.cc:654] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:58.926970 19091 log.cc:824] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:58.927835 19090 tablet_bootstrap.cc:492] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: No bootstrap required, opened a new log
I20260926 21:34:58.928118 19090 ts_tablet_manager.cc:1397] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 21:34:58.930051 19091 tablet_bootstrap.cc:492] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: No bootstrap required, opened a new log
I20260926 21:34:58.930330 19091 ts_tablet_manager.cc:1397] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 21:34:58.931722 19094 tablet_bootstrap.cc:492] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: Bootstrap starting.
I20260926 21:34:58.936039 19094 tablet_bootstrap.cc:654] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: Neither blocks nor log segments found. Creating new log.
I20260926 21:34:58.937127 19094 log.cc:824] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:34:58.938942 19090 raft_consensus.cc:348] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.939396 19090 raft_consensus.cc:374] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:58.939523 19090 raft_consensus.cc:729] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95e230854a400f869a1c9ffaaee067, State: Initialized, Role: FOLLOWER
I20260926 21:34:58.940121 19094 tablet_bootstrap.cc:492] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: No bootstrap required, opened a new log
I20260926 21:34:58.940171 19090 consensus_queue.cc:260] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.940392 19094 ts_tablet_manager.cc:1397] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 21:34:58.941308 18937 heartbeater.cc:503] Master 127.0.97.62:40047 was elected leader, sending a full tablet report...
I20260926 21:34:58.941991 19090 ts_tablet_manager.cc:1428] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 21:34:58.941047 19091 raft_consensus.cc:348] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.942777 19091 raft_consensus.cc:374] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:58.942922 19091 raft_consensus.cc:729] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87fca72c5f2244b4b90cb754ccf889ee, State: Initialized, Role: FOLLOWER
I20260926 21:34:58.943564 19091 consensus_queue.cc:260] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.944667 19071 heartbeater.cc:503] Master 127.0.97.62:40047 was elected leader, sending a full tablet report...
I20260926 21:34:58.945339 19091 ts_tablet_manager.cc:1428] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 21:34:58.950376 19094 raft_consensus.cc:348] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.950836 19094 raft_consensus.cc:374] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:34:58.950956 19094 raft_consensus.cc:729] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9550880bb94d769dbc8bd0ba93e91e, State: Initialized, Role: FOLLOWER
I20260926 21:34:58.951578 19094 consensus_queue.cc:260] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.953249 19094 ts_tablet_manager.cc:1428] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 21:34:58.972940 19097 raft_consensus.cc:482] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:34:58.973263 19097 raft_consensus.cc:504] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.975059 19097 leader_election.cc:288] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c95e230854a400f869a1c9ffaaee067 (127.0.97.2:41131), 6b9550880bb94d769dbc8bd0ba93e91e (127.0.97.1:44561)
I20260926 21:34:58.981298 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.975416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41131 (local address 127.0.97.3:51145)
0926 21:34:58.975746 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:58.975770 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:58.975887 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:58.976587 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:58.976591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:58.976615 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:58.976763 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.976770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.978042 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.978045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.979412 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.979424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.980532 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.980538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.980733 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.980757 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 21:34:58.980858 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 21:34:58.980998 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260926 21:34:58.981279 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.975866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51145 (local address 127.0.97.2:41131)
0926 21:34:58.976279 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:58.976287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:58.976308 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:34:58.976381 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:58.976386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:58.976459 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:58.976559 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:58.976919 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.977932 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.979572 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.980373 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.980523 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.980628 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:58.980895 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 21:34:58.981022 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260926 21:34:58.981734 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.975566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44965 (local address 127.0.97.1:44561)
0926 21:34:58.976104 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:58.976113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:58.976736 (+ 623us) server_negotiation.cc:407] Connection header received
0926 21:34:58.976798 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:58.976802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:58.976898 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:58.977022 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:34:58.977447 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.978323 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.980009 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.980803 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.980963 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.981068 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:58.981349 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 21:34:58.981480 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376}
I20260926 21:34:58.981771 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:58.975630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.97.3:44965)
0926 21:34:58.976458 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:58.976474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:34:58.976582 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:34:58.977034 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:34:58.977037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:34:58.977054 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:34:58.977298 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.977304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.978507 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.978511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.979854 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:34:58.979865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:34:58.981016 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:34:58.981021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:34:58.981194 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:58.981216 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:34:58.981324 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 21:34:58.981444 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":478,"threads_started":1}
I20260926 21:34:58.982764 18892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb2719448ce42949ce68f4dfe419bed" candidate_uuid: "87fca72c5f2244b4b90cb754ccf889ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95e230854a400f869a1c9ffaaee067" is_pre_election: true
I20260926 21:34:58.983196 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb2719448ce42949ce68f4dfe419bed" candidate_uuid: "87fca72c5f2244b4b90cb754ccf889ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" is_pre_election: true
I20260926 21:34:58.983361 18892 raft_consensus.cc:2454] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87fca72c5f2244b4b90cb754ccf889ee in term 0.
I20260926 21:34:58.983752 18758 raft_consensus.cc:2454] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87fca72c5f2244b4b90cb754ccf889ee in term 0.
I20260926 21:34:58.984144 18962 leader_election.cc:302] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c95e230854a400f869a1c9ffaaee067, 87fca72c5f2244b4b90cb754ccf889ee; no voters:
I20260926 21:34:58.984545 19097 raft_consensus.cc:2781] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:34:58.984699 19097 raft_consensus.cc:482] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:34:58.984810 19097 raft_consensus.cc:3037] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:58.987007 19097 raft_consensus.cc:504] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.987610 19097 leader_election.cc:288] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [CANDIDATE]: Term 1 election: Requested vote from peers 7c95e230854a400f869a1c9ffaaee067 (127.0.97.2:41131), 6b9550880bb94d769dbc8bd0ba93e91e (127.0.97.1:44561)
I20260926 21:34:58.987998 18892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb2719448ce42949ce68f4dfe419bed" candidate_uuid: "87fca72c5f2244b4b90cb754ccf889ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95e230854a400f869a1c9ffaaee067"
I20260926 21:34:58.988101 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb2719448ce42949ce68f4dfe419bed" candidate_uuid: "87fca72c5f2244b4b90cb754ccf889ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9550880bb94d769dbc8bd0ba93e91e"
I20260926 21:34:58.988225 18892 raft_consensus.cc:3037] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:58.988281 18758 raft_consensus.cc:3037] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:34:58.991473 18758 raft_consensus.cc:2454] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87fca72c5f2244b4b90cb754ccf889ee in term 1.
I20260926 21:34:58.991473 18892 raft_consensus.cc:2454] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87fca72c5f2244b4b90cb754ccf889ee in term 1.
I20260926 21:34:58.991887 18962 leader_election.cc:302] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c95e230854a400f869a1c9ffaaee067, 87fca72c5f2244b4b90cb754ccf889ee; no voters:
I20260926 21:34:58.992156 19097 raft_consensus.cc:2781] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:34:58.992576 19097 raft_consensus.cc:686] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 1 LEADER]: Becoming Leader. State: Replica: 87fca72c5f2244b4b90cb754ccf889ee, State: Running, Role: LEADER
I20260926 21:34:58.993067 19097 consensus_queue.cc:237] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } }
I20260926 21:34:58.997679 18614 catalog_manager.cc:5495] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 87fca72c5f2244b4b90cb754ccf889ee (127.0.97.3). New cstate: current_term: 1 leader_uuid: "87fca72c5f2244b4b90cb754ccf889ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fca72c5f2244b4b90cb754ccf889ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 } health_report { overall_health: UNKNOWN } } }
W20260926 21:34:59.010018 18804 tablet.cc:2369] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:34:59.031515 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.026338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.0.97.3:41299)
0926 21:34:59.026584 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:59.026595 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:59.026613 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:34:59.026761 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:59.026765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:59.026846 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:59.026965 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:34:59.027299 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:59.028020 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:59.029387 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:59.029786 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:59.029878 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:59.030339 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:34:59.030885 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:34:59.031011 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:59.031159 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 21:34:59.031226 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260926 21:34:59.032624 18938 tablet.cc:2369] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:34:59.040340 18892 raft_consensus.cc:1261] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 1 FOLLOWER]: Refusing update from remote peer 87fca72c5f2244b4b90cb754ccf889ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:34:59.040340 18758 raft_consensus.cc:1261] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 1 FOLLOWER]: Refusing update from remote peer 87fca72c5f2244b4b90cb754ccf889ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:34:59.041339 19097 consensus_queue.cc:1035] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9550880bb94d769dbc8bd0ba93e91e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:34:59.041956 19100 consensus_queue.cc:1035] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c95e230854a400f869a1c9ffaaee067" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:34:59.050644 19072 tablet.cc:2369] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:34:59.056229 19103 mvcc.cc:205] Tried to move back new op lower bound from 7333718012059074560 to 7333718011879223296. Current Snapshot: MvccSnapshot[applied={T|T < 7333718012059074560}]
I20260926 21:34:59.071605 18614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:34:59.071866 18614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:34:59.075488 18614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83df1682407746a8a8307e510895faf4: Sending DeleteTablet for 3 replicas of tablet 7eb2719448ce42949ce68f4dfe419bed
I20260926 21:34:59.076732 18872 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb2719448ce42949ce68f4dfe419bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35462
I20260926 21:34:59.076738 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb2719448ce42949ce68f4dfe419bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44636
I20260926 21:34:59.078299 19006 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb2719448ce42949ce68f4dfe419bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:34:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37936
I20260926 21:34:59.083213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 18675
I20260926 21:34:59.086972 19114 tablet_replica.cc:330] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: stopping tablet replica
I20260926 21:34:59.087478 19114 raft_consensus.cc:2229] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:34:59.088040 19114 raft_consensus.cc:2258] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:59.092129 19114 ts_tablet_manager.cc:1905] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:59.099932 19114 ts_tablet_manager.cc:1918] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:59.100236 19114 log.cc:1196] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/7eb2719448ce42949ce68f4dfe419bed
I20260926 21:34:59.100916 19114 ts_tablet_manager.cc:1939] T 7eb2719448ce42949ce68f4dfe419bed P 87fca72c5f2244b4b90cb754ccf889ee: Deleting consensus metadata
I20260926 21:34:59.103044 18598 catalog_manager.cc:4841] TS 87fca72c5f2244b4b90cb754ccf889ee (127.0.97.3:41299): tablet 7eb2719448ce42949ce68f4dfe419bed (table test-table [id=55266fc2bbf74160b1f7d71e28283914]) successfully deleted
I20260926 21:34:59.111902 19113 tablet_replica.cc:330] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: stopping tablet replica
I20260926 21:34:59.116104 19113 raft_consensus.cc:2229] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:34:59.116850 19113 raft_consensus.cc:2258] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:59.120832 19113 ts_tablet_manager.cc:1905] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:34:59 dist-test-slave-87bt krb5kdc[18569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:34:59.131126 19113 ts_tablet_manager.cc:1918] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:34:59.131932 19112 tablet_replica.cc:330] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: stopping tablet replica
I20260926 21:34:59.132514 19112 raft_consensus.cc:2229] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:34:59.132990 19113 log.cc:1196] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/7eb2719448ce42949ce68f4dfe419bed
I20260926 21:34:59.133260 19112 raft_consensus.cc:2258] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:34:59.133668 19113 ts_tablet_manager.cc:1939] T 7eb2719448ce42949ce68f4dfe419bed P 7c95e230854a400f869a1c9ffaaee067: Deleting consensus metadata
I20260926 21:34:59.135663 18601 catalog_manager.cc:4841] TS 7c95e230854a400f869a1c9ffaaee067 (127.0.97.2:41131): tablet 7eb2719448ce42949ce68f4dfe419bed (table test-table [id=55266fc2bbf74160b1f7d71e28283914]) successfully deleted
I20260926 21:34:59.136198 19112 ts_tablet_manager.cc:1905] T 7eb2719448ce42949ce68f4dfe419bed P 6b9550880bb94d769dbc8bd0ba93e91e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:34:59.140107 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.085008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.0.97.1:44561)
0926 21:34:59.085254 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:59.085264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:59.108879 (+ 23615us) server_negotiation.cc:407] Connection header received
0926 21:34:59.109379 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:59.109385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:59.109513 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:59.109583 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:59.112201 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:59.113449 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:59.117311 (+ 3862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:59.118015 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:59.118126 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:59.131989 (+ 13863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:59.132011 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:59.132030 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:59.132078 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:59.136338 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:59.137326 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:59.137336 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:59.137340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:59.137527 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:59.138124 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:59.138130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:59.138133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:59.138705 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:59.139076 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:59.139427 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 21:34:59.139692 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 21:34:59.140877 18798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:59.232667 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 18675
I20260926 21:34:59.251178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 18809
W20260926 21:34:59.251235 18600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44561 (error 108)
I20260926 21:34:59.253575 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.253102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44640)
0926 21:34:59.253274 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:59.253392 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 21:34:59.253845 18600 catalog_manager.cc:4584] TS 6b9550880bb94d769dbc8bd0ba93e91e (127.0.97.1:44561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7eb2719448ce42949ce68f4dfe419bed: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Sep 26 21:34:59 dist-test-slave-87bt krb5kdc[18569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:34:59.269124 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.252200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.0.97.2:41131)
0926 21:34:59.252407 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:59.252413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:59.252553 (+ 140us) server_negotiation.cc:407] Connection header received
0926 21:34:59.252728 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:59.252734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:59.252839 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:59.252974 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:59.254497 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:59.255653 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:59.257628 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:59.258021 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:59.258121 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:59.263212 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:59.263227 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:59.263243 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:59.263277 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:59.266214 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:59.267038 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:59.267046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:59.267048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:59.267177 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:59.267557 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:59.267563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:59.267568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:59.267947 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:59.268245 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:59.268551 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 21:34:59.268783 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:34:59.269747 18932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:59.307848 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.307291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44642)
0926 21:34:59.307538 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:59.307661 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 21:34:59.309067 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.308624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44658)
0926 21:34:59.308800 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:59.308893 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:34:59.361191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 18809
I20260926 21:34:59.369930 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.369316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44668)
0926 21:34:59.369589 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:59.369707 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 21:34:59.371218 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.370769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44672)
0926 21:34:59.370952 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:59.371046 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 21:34:59.379570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 18943
Sep 26 21:34:59 dist-test-slave-87bt krb5kdc[18569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:34:59.396436 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.380365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.0.97.3:41299)
0926 21:34:59.380550 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 21:34:59.380556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:34:59.380614 (+ 58us) server_negotiation.cc:407] Connection header received
0926 21:34:59.380830 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:34:59.380835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:34:59.380920 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:34:59.381034 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:34:59.382332 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:59.383219 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:59.384504 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:34:59.384981 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:34:59.385119 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:34:59.389470 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:34:59.389482 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:34:59.389498 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:34:59.389536 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:34:59.393379 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:59.394102 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:59.394113 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:59.394122 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:59.394290 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:34:59.394716 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:34:59.394724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:34:59.394731 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:34:59.395120 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:34:59.395527 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0926 21:34:59.395835 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 21:34:59.396065 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:34:59.397133 19066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:59.451789 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.451352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44684)
0926 21:34:59.451557 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:59.451645 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:34:59.452977 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.452525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44696)
0926 21:34:59.452754 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:59.452830 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 21:34:59.521675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 18943
I20260926 21:34:59.542083 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 18582
I20260926 21:34:59.543354 18644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:34:59.656303 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:34:59.655292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44561 (local address 127.0.0.1:44706)
0926 21:34:59.655527 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 21:34:59.655638 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 21:34:59.656454 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 18582
2026-09-26T21:34:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4362 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T21:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:34:59Z Disabled control of system clock
I20260926 21:34:59.732614 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37355
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:35:00.066965 19124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:00.067287 19124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:00.067467 19124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:00.074810 19124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:00.074920 19124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:00.075023 19124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:00.075148 19124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:00.075219 19124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:00.084468 19124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:00.085856 19124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:00.087663 19124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:00.099409 19130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:00.099499 19132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:00.099490 19129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:00.100620 19124 server_base.cc:1034] running on GCE node
I20260926 21:35:00.101516 19124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:00.103102 19124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:00.104341 19124 hybrid_clock.cc:648] HybridClock initialized: now 1790458500104290 us; error 46 us; skew 500 ppm
I20260926 21:35:00.104697 19124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:35:00.107132 19124 webserver.cc:460] Webserver started at http://127.0.97.62:38131/ using document root <none> and password file <none>
I20260926 21:35:00.107831 19124 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:00.107967 19124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:00.108295 19124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:00.111347 19124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "f0da5103bf034ab38eec55a0bd030a31"
format_stamp: "Formatted at 2026-09-26 21:35:00 on dist-test-slave-87bt"
server_key: "b14b18245dfa16388a20855bd7926c87"
server_key_iv: "c0c3aa4ed6ddfa3f7c0861446daeeb76"
server_key_version: "encryptionkey@0"
I20260926 21:35:00.112219 19124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "f0da5103bf034ab38eec55a0bd030a31"
format_stamp: "Formatted at 2026-09-26 21:35:00 on dist-test-slave-87bt"
server_key: "b14b18245dfa16388a20855bd7926c87"
server_key_iv: "c0c3aa4ed6ddfa3f7c0861446daeeb76"
server_key_version: "encryptionkey@0"
I20260926 21:35:00.117527 19124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:35:00.121117 19138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:00.122630 19124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:00.122836 19124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "f0da5103bf034ab38eec55a0bd030a31"
format_stamp: "Formatted at 2026-09-26 21:35:00 on dist-test-slave-87bt"
server_key: "b14b18245dfa16388a20855bd7926c87"
server_key_iv: "c0c3aa4ed6ddfa3f7c0861446daeeb76"
server_key_version: "encryptionkey@0"
I20260926 21:35:00.123093 19124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:00.139302 19124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:00.140210 19124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:00.140519 19124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:00.159107 19124 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37355
I20260926 21:35:00.159114 19189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37355 every 8 connection(s)
I20260926 21:35:00.160518 19124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:00.164183 19190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:00.166538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 19124
I20260926 21:35:00.166713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:00.167254 388 external_mini_cluster.cc:1428] Setting key 9b61320e77d03c12a00aaf71fdb846ad
I20260926 21:35:00.176167 19190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31: Bootstrap starting.
I20260926 21:35:00.179308 19190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:00.180406 19190 log.cc:824] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:00.182246 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:00.169243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.0.97.62:37355)
0926 21:35:00.170362 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:00.170377 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:00.170504 (+ 127us) server_negotiation.cc:407] Connection header received
0926 21:35:00.172525 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:00.172550 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:00.173113 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:00.173897 (+ 784us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:00.174614 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:00.175840 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:00.177318 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:00.178045 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:00.178253 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:00.179502 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:00.179513 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:00.179533 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:00.180111 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:00.180203 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:00.180226 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:00.180360 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:00.180587 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 21:35:00.180918 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":369,"threads_started":1}
I20260926 21:35:00.183830 19190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31: No bootstrap required, opened a new log
I20260926 21:35:00.191567 19190 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da5103bf034ab38eec55a0bd030a31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37355 } }
I20260926 21:35:00.192008 19190 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:00.192128 19190 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0da5103bf034ab38eec55a0bd030a31, State: Initialized, Role: FOLLOWER
I20260926 21:35:00.192788 19190 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da5103bf034ab38eec55a0bd030a31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37355 } }
I20260926 21:35:00.193073 19190 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:00.193249 19190 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:00.193439 19190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:00.195734 19190 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da5103bf034ab38eec55a0bd030a31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37355 } }
I20260926 21:35:00.196316 19190 leader_election.cc:302] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0da5103bf034ab38eec55a0bd030a31; no voters:
I20260926 21:35:00.197001 19190 leader_election.cc:288] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:00.197214 19195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:00.198496 19195 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [term 1 LEADER]: Becoming Leader. State: Replica: f0da5103bf034ab38eec55a0bd030a31, State: Running, Role: LEADER
I20260926 21:35:00.199041 19195 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da5103bf034ab38eec55a0bd030a31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37355 } }
I20260926 21:35:00.199426 19190 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:00.202080 19197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0da5103bf034ab38eec55a0bd030a31. Latest consensus state: current_term: 1 leader_uuid: "f0da5103bf034ab38eec55a0bd030a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da5103bf034ab38eec55a0bd030a31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37355 } } }
I20260926 21:35:00.202522 19197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:00.202199 19196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0da5103bf034ab38eec55a0bd030a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da5103bf034ab38eec55a0bd030a31" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37355 } } }
I20260926 21:35:00.202673 19196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:00.205228 19201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:00.211971 19201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:00.221134 19201 catalog_manager.cc:1347] Generated new cluster ID: 5bf9859fb78b409ca745b62f951ec7b4
I20260926 21:35:00.221283 19201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:00.245950 19201 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:00.247632 19201 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:00.255267 19201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31: Generated new TSK 0
I20260926 21:35:00.256381 19201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:00.271013 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37355
--builtin_ntp_servers=127.0.97.20:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:35:00.608233 19214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:00.608587 19214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:00.608680 19214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:00.616195 19214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:00.616317 19214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:00.616425 19214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:35:00.626121 19214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:00.627796 19214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:00.629662 19214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:00.643867 19220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:00.644006 19219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:00.643965 19222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:00.644990 19214 server_base.cc:1034] running on GCE node
I20260926 21:35:00.645661 19214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:00.646845 19214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:00.648108 19214 hybrid_clock.cc:648] HybridClock initialized: now 1790458500648047 us; error 65 us; skew 500 ppm
I20260926 21:35:00.648474 19214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:35:00.650993 19214 webserver.cc:460] Webserver started at http://127.0.97.1:38915/ using document root <none> and password file <none>
I20260926 21:35:00.651759 19214 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:00.651897 19214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:00.652237 19214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:00.655483 19214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "3d591938a2634e049da399b7d7275005"
format_stamp: "Formatted at 2026-09-26 21:35:00 on dist-test-slave-87bt"
server_key: "6af495c8d8f6395147cb664cb621b930"
server_key_iv: "5f5981ce41935caa1a2088d28d73b06b"
server_key_version: "encryptionkey@0"
I20260926 21:35:00.656303 19214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3d591938a2634e049da399b7d7275005"
format_stamp: "Formatted at 2026-09-26 21:35:00 on dist-test-slave-87bt"
server_key: "6af495c8d8f6395147cb664cb621b930"
server_key_iv: "5f5981ce41935caa1a2088d28d73b06b"
server_key_version: "encryptionkey@0"
I20260926 21:35:00.661408 19214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:35:00.664816 19228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:00.666144 19214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:35:00.666312 19214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "3d591938a2634e049da399b7d7275005"
format_stamp: "Formatted at 2026-09-26 21:35:00 on dist-test-slave-87bt"
server_key: "6af495c8d8f6395147cb664cb621b930"
server_key_iv: "5f5981ce41935caa1a2088d28d73b06b"
server_key_version: "encryptionkey@0"
I20260926 21:35:00.666584 19214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:00.679756 19214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:00.680574 19214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:00.680910 19214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:00.682015 19214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:00.683548 19214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:00.683637 19214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:00.683719 19214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:00.683773 19214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:00.715971 19214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36657
I20260926 21:35:00.715991 19340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36657 every 8 connection(s)
I20260926 21:35:00.717307 19214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:35:00.726399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 19214
I20260926 21:35:00.726608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:35:00.727144 388 external_mini_cluster.cc:1428] Setting key 40debfe2f2dc137b6de14c669c0b931a
I20260926 21:35:00.730300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37355
--builtin_ntp_servers=127.0.97.20:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:35:00.730546 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:00.719809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51137 (local address 127.0.97.62:37355)
0926 21:35:00.720234 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:00.720239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:00.721487 (+ 1248us) server_negotiation.cc:407] Connection header received
0926 21:35:00.723342 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:00.723346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:00.723414 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:00.723511 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:00.724531 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:00.725252 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:00.727376 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:00.727834 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:00.727994 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:00.729606 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:00.729612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:00.729617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:00.729638 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:00.729688 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:00.729707 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:00.729803 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:00.730018 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 21:35:00.730135 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":222,"threads_started":1}
I20260926 21:35:00.731588 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:00.720427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37355 (local address 127.0.97.1:51137)
0926 21:35:00.721323 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:00.721357 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:00.723087 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:00.723728 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:00.723737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:00.723826 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:00.724386 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:00.724400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:00.725390 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:00.725393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:00.727196 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:00.727213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:00.728111 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:00.728120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:00.728435 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:00.729258 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:35:00.729262 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:00.729314 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:35:00.729398 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:00.729808 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:00.729816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:00.729822 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:35:00.729970 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 21:35:00.730303 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":284,"threads_started":1}
I20260926 21:35:00.733249 19341 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37355
I20260926 21:35:00.733600 19341 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:00.734601 19341 heartbeater.cc:511] Master 127.0.97.62:37355 requested a full tablet report, sending...
I20260926 21:35:00.736829 19155 ts_manager.cc:194] Registered new tserver with Master: 3d591938a2634e049da399b7d7275005 (127.0.97.1:36657)
I20260926 21:35:00.738328 19155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:51137
W20260926 21:35:01.069429 19345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:01.069842 19345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:01.069960 19345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:01.077448 19345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:01.077559 19345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:01.077665 19345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:35:01.086859 19345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:01.088109 19345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:01.089774 19345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:01.103150 19350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:01.103235 19353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:01.103257 19351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:01.104017 19345 server_base.cc:1034] running on GCE node
I20260926 21:35:01.104692 19345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:01.105769 19345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:01.107002 19345 hybrid_clock.cc:648] HybridClock initialized: now 1790458501106945 us; error 48 us; skew 500 ppm
I20260926 21:35:01.107339 19345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:35:01.109717 19345 webserver.cc:460] Webserver started at http://127.0.97.2:34559/ using document root <none> and password file <none>
I20260926 21:35:01.110502 19345 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:01.110630 19345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:01.111019 19345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:01.113979 19345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "cfc21149b1d9454d924948d17e6cbe06"
format_stamp: "Formatted at 2026-09-26 21:35:01 on dist-test-slave-87bt"
server_key: "7afaa7d09860abf502dfc014d00d961f"
server_key_iv: "6de3441b519adf44c33a6328283df312"
server_key_version: "encryptionkey@0"
I20260926 21:35:01.114866 19345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cfc21149b1d9454d924948d17e6cbe06"
format_stamp: "Formatted at 2026-09-26 21:35:01 on dist-test-slave-87bt"
server_key: "7afaa7d09860abf502dfc014d00d961f"
server_key_iv: "6de3441b519adf44c33a6328283df312"
server_key_version: "encryptionkey@0"
I20260926 21:35:01.120188 19345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 21:35:01.123795 19359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:01.125234 19345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:35:01.125427 19345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "cfc21149b1d9454d924948d17e6cbe06"
format_stamp: "Formatted at 2026-09-26 21:35:01 on dist-test-slave-87bt"
server_key: "7afaa7d09860abf502dfc014d00d961f"
server_key_iv: "6de3441b519adf44c33a6328283df312"
server_key_version: "encryptionkey@0"
I20260926 21:35:01.125656 19345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:01.159199 19345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:01.160053 19345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:01.160355 19345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:01.161366 19345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:01.162938 19345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:01.163060 19345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:01.163199 19345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:01.163286 19345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:01.198208 19345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37179
I20260926 21:35:01.198263 19471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37179 every 8 connection(s)
I20260926 21:35:01.199843 19345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:35:01.207167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 19345
I20260926 21:35:01.207358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:35:01.207784 388 external_mini_cluster.cc:1428] Setting key 50d08dfab24a81df28f5ea3efa27bc35
I20260926 21:35:01.210634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37355
--builtin_ntp_servers=127.0.97.20:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:35:01.215224 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.204362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51109 (local address 127.0.97.62:37355)
0926 21:35:01.204568 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.204573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.206236 (+ 1663us) server_negotiation.cc:407] Connection header received
0926 21:35:01.208361 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.208367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.208466 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.208592 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:01.209743 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.210552 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.212272 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.212680 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.212826 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.214288 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:01.214293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:01.214296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:01.214325 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:01.214358 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:01.214371 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:01.214525 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.214789 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.214896 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:35:01.216358 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.204933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37355 (local address 127.0.97.2:51109)
0926 21:35:01.206058 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:01.206104 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:01.208050 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:01.208849 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:01.208858 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:01.208960 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:01.209571 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.209590 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.210728 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.210732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.212111 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.212124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.212928 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.212935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.213185 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.213940 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:35:01.213943 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:01.213987 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:35:01.214080 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:01.214532 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:01.214544 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:01.214556 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 21:35:01.214764 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 21:35:01.215215 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":386,"threads_started":1}
I20260926 21:35:01.217938 19472 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37355
I20260926 21:35:01.218326 19472 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:01.219377 19472 heartbeater.cc:511] Master 127.0.97.62:37355 requested a full tablet report, sending...
I20260926 21:35:01.221195 19155 ts_manager.cc:194] Registered new tserver with Master: cfc21149b1d9454d924948d17e6cbe06 (127.0.97.2:37179)
I20260926 21:35:01.222514 19155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:51109
W20260926 21:35:01.545130 19475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:01.545446 19475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:01.545536 19475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:01.553054 19475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:01.553171 19475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:01.553278 19475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:35:01.562955 19475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:01.564340 19475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:01.566138 19475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:01.579710 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:01.579774 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:01.579735 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:01.580470 19475 server_base.cc:1034] running on GCE node
I20260926 21:35:01.581072 19475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:01.581982 19475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:01.583236 19475 hybrid_clock.cc:648] HybridClock initialized: now 1790458501583184 us; error 40 us; skew 500 ppm
I20260926 21:35:01.583583 19475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:35:01.586023 19475 webserver.cc:460] Webserver started at http://127.0.97.3:46607/ using document root <none> and password file <none>
I20260926 21:35:01.586995 19475 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:01.587136 19475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:01.587508 19475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:01.591923 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "14550af5415642659c5f4365c23e344a"
format_stamp: "Formatted at 2026-09-26 21:35:01 on dist-test-slave-87bt"
server_key: "c02fe2e2743c60ae7734fa1fa4c9b227"
server_key_iv: "4aa63f6463739fc2c7f7f7adbb995248"
server_key_version: "encryptionkey@0"
I20260926 21:35:01.593113 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "14550af5415642659c5f4365c23e344a"
format_stamp: "Formatted at 2026-09-26 21:35:01 on dist-test-slave-87bt"
server_key: "c02fe2e2743c60ae7734fa1fa4c9b227"
server_key_iv: "4aa63f6463739fc2c7f7f7adbb995248"
server_key_version: "encryptionkey@0"
I20260926 21:35:01.600477 19475 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 21:35:01.604184 19489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:01.605718 19475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:35:01.605887 19475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "14550af5415642659c5f4365c23e344a"
format_stamp: "Formatted at 2026-09-26 21:35:01 on dist-test-slave-87bt"
server_key: "c02fe2e2743c60ae7734fa1fa4c9b227"
server_key_iv: "4aa63f6463739fc2c7f7f7adbb995248"
server_key_version: "encryptionkey@0"
I20260926 21:35:01.606122 19475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:01.637251 19475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:01.638160 19475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:01.638501 19475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:01.639740 19475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:01.641265 19475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:01.641355 19475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:01.641436 19475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:01.641490 19475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:01.675118 19475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37939
I20260926 21:35:01.675207 19601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37939 every 8 connection(s)
I20260926 21:35:01.676445 19475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:35:01.678715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 19475
I20260926 21:35:01.678900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:35:01.679387 388 external_mini_cluster.cc:1428] Setting key ea05c8c85e164a845d1ed0358ee3980d
I20260926 21:35:01.689231 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.679036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52721 (local address 127.0.97.62:37355)
0926 21:35:01.679275 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.679282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.681004 (+ 1722us) server_negotiation.cc:407] Connection header received
0926 21:35:01.682875 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.682880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.682964 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.683089 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:01.684163 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.684889 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.686444 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.686867 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.687003 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.688371 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:01.688377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:01.688382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:01.688409 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:01.688439 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:01.688450 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:01.688546 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.688803 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.688900 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 21:35:01.690207 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.679779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37355 (local address 127.0.97.3:52721)
0926 21:35:01.680842 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:01.680885 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:01.682604 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:01.683326 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:01.683335 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:01.683428 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:01.683999 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.684015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.685042 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.685055 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.686278 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.686289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.687091 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.687098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.687332 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.688048 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:35:01.688051 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:01.688095 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:35:01.688171 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:01.688544 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:01.688564 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:01.688574 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:35:01.688794 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 21:35:01.689184 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":369,"threads_started":1}
I20260926 21:35:01.691612 19602 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37355
I20260926 21:35:01.691913 19602 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:01.692655 19602 heartbeater.cc:511] Master 127.0.97.62:37355 requested a full tablet report, sending...
I20260926 21:35:01.694195 19155 ts_manager.cc:194] Registered new tserver with Master: 14550af5415642659c5f4365c23e344a (127.0.97.3:37939)
I20260926 21:35:01.694905 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:35:01.695331 19155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52721
I20260926 21:35:01.705386 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.699299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.97.62:37355)
0926 21:35:01.699492 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.699500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.699906 (+ 406us) server_negotiation.cc:407] Connection header received
0926 21:35:01.700117 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.700122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.700216 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.700333 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:01.700811 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.701488 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.702919 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.703417 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.703543 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.704656 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:01.704669 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:01.704673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:01.704689 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:01.704720 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:01.704731 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:01.704881 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.705065 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.705118 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 21:35:01.709090 19155 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:35:01.715966 19155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:35:01.742156 19341 heartbeater.cc:503] Master 127.0.97.62:37355 was elected leader, sending a full tablet report...
I20260926 21:35:01.743260 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.728710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37179 (local address 127.0.0.1:41548)
0926 21:35:01.730163 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:01.730180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:01.730278 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:01.735841 (+ 5563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:01.735851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:01.735877 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:01.736315 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.736324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.738024 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.738032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.740134 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.740149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.742385 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.742391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.742601 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.742632 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:35:01.742721 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 21:35:01.742853 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":56,"thread_start_us":5911,"threads_started":1}
I20260926 21:35:01.743703 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.730251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.97.3:37939)
0926 21:35:01.731060 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.731070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.734998 (+ 3928us) server_negotiation.cc:407] Connection header received
0926 21:35:01.735169 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.735175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.735355 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.735553 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:01.736288 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.737962 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.740306 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.741574 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.741711 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.743080 (+ 1369us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.743156 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.743325 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":410,"threads_started":1}
I20260926 21:35:01.744074 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.729589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37939 (local address 127.0.0.1:49044)
0926 21:35:01.734866 (+ 5277us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:01.734885 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:01.735027 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:01.735567 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:01.735575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:01.735606 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:01.736023 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.736035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.738138 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.738144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.740126 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.740138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.742145 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.742152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.742366 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.742444 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 21:35:01.742613 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 21:35:01.742811 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5160,"mutex_wait_us":2,"thread_start_us":886,"threads_started":1}
I20260926 21:35:01.744511 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.728490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36657 (local address 127.0.0.1:48232)
0926 21:35:01.729337 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:01.729379 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:01.729553 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:01.736895 (+ 7342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:01.736899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:01.736925 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:01.737286 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.737297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.739485 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.739490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.741666 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.741681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.743736 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.743743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.743924 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.743956 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:35:01.744032 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 21:35:01.744186 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":1,"thread_start_us":404,"threads_started":1}
I20260926 21:35:01.744635 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.729357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.0.97.2:37179)
0926 21:35:01.734850 (+ 5493us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.734862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.734901 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:35:01.735036 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.735060 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.735393 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.735632 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:01.736492 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.737892 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.740365 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.741861 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.741993 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.742251 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.744130 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.744308 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5349,"thread_start_us":5264,"threads_started":1}
I20260926 21:35:01.745635 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.728800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.0.97.1:36657)
0926 21:35:01.736282 (+ 7482us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.736290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.736318 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:35:01.736417 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.736424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.736645 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.737484 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:01.737700 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.739328 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.741903 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.743505 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.744853 (+ 1348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.745075 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.745125 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.745245 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7318,"thread_start_us":7185,"threads_started":1}
I20260926 21:35:01.748172 19407 tablet_service.cc:1467] Processing CreateTablet for tablet d726e72f34db49ffbe24342d3143ac6b (DEFAULT_TABLE table=test-table [id=364b76d7814e47e78787233c2517b993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:01.748369 19537 tablet_service.cc:1467] Processing CreateTablet for tablet d726e72f34db49ffbe24342d3143ac6b (DEFAULT_TABLE table=test-table [id=364b76d7814e47e78787233c2517b993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:01.748903 19276 tablet_service.cc:1467] Processing CreateTablet for tablet d726e72f34db49ffbe24342d3143ac6b (DEFAULT_TABLE table=test-table [id=364b76d7814e47e78787233c2517b993]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:01.749509 19407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d726e72f34db49ffbe24342d3143ac6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:01.749867 19537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d726e72f34db49ffbe24342d3143ac6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:01.750195 19276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d726e72f34db49ffbe24342d3143ac6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:01.761763 19620 tablet_bootstrap.cc:492] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: Bootstrap starting.
I20260926 21:35:01.764385 19621 tablet_bootstrap.cc:492] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: Bootstrap starting.
I20260926 21:35:01.765370 19620 tablet_bootstrap.cc:654] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:01.766757 19620 log.cc:824] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:01.767212 19622 tablet_bootstrap.cc:492] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: Bootstrap starting.
I20260926 21:35:01.768196 19621 tablet_bootstrap.cc:654] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:01.769397 19621 log.cc:824] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:01.770658 19620 tablet_bootstrap.cc:492] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: No bootstrap required, opened a new log
I20260926 21:35:01.770919 19622 tablet_bootstrap.cc:654] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:01.770941 19620 ts_tablet_manager.cc:1397] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 21:35:01.772157 19622 log.cc:824] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:01.772387 19621 tablet_bootstrap.cc:492] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: No bootstrap required, opened a new log
I20260926 21:35:01.772660 19621 ts_tablet_manager.cc:1397] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 21:35:01.775178 19622 tablet_bootstrap.cc:492] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: No bootstrap required, opened a new log
I20260926 21:35:01.775424 19622 ts_tablet_manager.cc:1397] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 21:35:01.778740 19620 raft_consensus.cc:348] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.779109 19620 raft_consensus.cc:374] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:01.779234 19620 raft_consensus.cc:729] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc21149b1d9454d924948d17e6cbe06, State: Initialized, Role: FOLLOWER
I20260926 21:35:01.779989 19620 consensus_queue.cc:260] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.781940 19472 heartbeater.cc:503] Master 127.0.97.62:37355 was elected leader, sending a full tablet report...
I20260926 21:35:01.782658 19620 ts_tablet_manager.cc:1428] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 21:35:01.783228 19621 raft_consensus.cc:348] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.783655 19621 raft_consensus.cc:374] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:01.783751 19621 raft_consensus.cc:729] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d591938a2634e049da399b7d7275005, State: Initialized, Role: FOLLOWER
I20260926 21:35:01.784354 19621 consensus_queue.cc:260] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.785753 19622 raft_consensus.cc:348] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.786141 19621 ts_tablet_manager.cc:1428] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 21:35:01.786183 19622 raft_consensus.cc:374] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:01.786283 19622 raft_consensus.cc:729] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14550af5415642659c5f4365c23e344a, State: Initialized, Role: FOLLOWER
I20260926 21:35:01.786891 19622 consensus_queue.cc:260] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.788125 19602 heartbeater.cc:503] Master 127.0.97.62:37355 was elected leader, sending a full tablet report...
I20260926 21:35:01.788468 19622 ts_tablet_manager.cc:1428] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 21:35:01.828156 19626 raft_consensus.cc:482] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:35:01.828516 19626 raft_consensus.cc:504] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.830577 19626 leader_election.cc:288] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d591938a2634e049da399b7d7275005 (127.0.97.1:36657), 14550af5415642659c5f4365c23e344a (127.0.97.3:37939)
I20260926 21:35:01.837713 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.830846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36657 (local address 127.0.97.2:53403)
0926 21:35:01.831902 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:01.831947 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:01.832098 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:01.832458 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:01.832466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:01.832495 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:01.832843 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.832851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.833865 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.833873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.835585 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.835602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.836832 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.836839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.837043 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.837080 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 21:35:01.837181 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 21:35:01.837350 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":69,"thread_start_us":703,"threads_started":1}
I20260926 21:35:01.837807 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.831178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34473 (local address 127.0.97.3:37939)
0926 21:35:01.831435 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.831444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.831973 (+ 529us) server_negotiation.cc:407] Connection header received
0926 21:35:01.832167 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.832174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.832250 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.832327 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:01.833090 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.833809 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.835840 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.836639 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.836801 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.836909 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.837361 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.837495 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:35:01.838224 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.830816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53403 (local address 127.0.97.1:36657)
0926 21:35:01.831036 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.831045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.832069 (+ 1024us) server_negotiation.cc:407] Connection header received
0926 21:35:01.832232 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.832235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.832329 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.832420 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:01.833008 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.833726 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.835807 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.836606 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.836743 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.836860 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.837275 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.837508 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:35:01.838634 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.830814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37939 (local address 127.0.97.2:34473)
0926 21:35:01.831853 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:01.831869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:01.831989 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:01.832375 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:01.832381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:01.832409 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:01.832915 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.832947 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.833939 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.833942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.835613 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:01.835626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.837023 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.837028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:01.837203 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.837230 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 21:35:01.838008 (+ 778us) client_negotiation.cc:240] Negotiation successful
0926 21:35:01.838154 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":72,"spinlock_wait_cycles":33152,"thread_start_us":616,"threads_started":1}
I20260926 21:35:01.839826 19557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d726e72f34db49ffbe24342d3143ac6b" candidate_uuid: "cfc21149b1d9454d924948d17e6cbe06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14550af5415642659c5f4365c23e344a" is_pre_election: true
I20260926 21:35:01.839869 19296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d726e72f34db49ffbe24342d3143ac6b" candidate_uuid: "cfc21149b1d9454d924948d17e6cbe06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d591938a2634e049da399b7d7275005" is_pre_election: true
I20260926 21:35:01.840447 19296 raft_consensus.cc:2454] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc21149b1d9454d924948d17e6cbe06 in term 0.
I20260926 21:35:01.840443 19557 raft_consensus.cc:2454] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc21149b1d9454d924948d17e6cbe06 in term 0.
I20260926 21:35:01.841207 19362 leader_election.cc:302] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14550af5415642659c5f4365c23e344a, 3d591938a2634e049da399b7d7275005, cfc21149b1d9454d924948d17e6cbe06; no voters:
I20260926 21:35:01.841693 19626 raft_consensus.cc:2781] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:35:01.841885 19626 raft_consensus.cc:482] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:35:01.842020 19626 raft_consensus.cc:3037] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:01.844347 19626 raft_consensus.cc:504] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.845036 19626 leader_election.cc:288] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [CANDIDATE]: Term 1 election: Requested vote from peers 3d591938a2634e049da399b7d7275005 (127.0.97.1:36657), 14550af5415642659c5f4365c23e344a (127.0.97.3:37939)
I20260926 21:35:01.845572 19557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d726e72f34db49ffbe24342d3143ac6b" candidate_uuid: "cfc21149b1d9454d924948d17e6cbe06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14550af5415642659c5f4365c23e344a"
I20260926 21:35:01.845587 19296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d726e72f34db49ffbe24342d3143ac6b" candidate_uuid: "cfc21149b1d9454d924948d17e6cbe06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d591938a2634e049da399b7d7275005"
I20260926 21:35:01.845786 19557 raft_consensus.cc:3037] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:01.845808 19296 raft_consensus.cc:3037] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:01.848035 19557 raft_consensus.cc:2454] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc21149b1d9454d924948d17e6cbe06 in term 1.
I20260926 21:35:01.848029 19296 raft_consensus.cc:2454] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc21149b1d9454d924948d17e6cbe06 in term 1.
I20260926 21:35:01.848500 19361 leader_election.cc:302] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14550af5415642659c5f4365c23e344a, cfc21149b1d9454d924948d17e6cbe06; no voters:
I20260926 21:35:01.848804 19626 raft_consensus.cc:2781] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:01.849318 19626 raft_consensus.cc:686] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 1 LEADER]: Becoming Leader. State: Replica: cfc21149b1d9454d924948d17e6cbe06, State: Running, Role: LEADER
I20260926 21:35:01.849822 19626 consensus_queue.cc:237] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } }
I20260926 21:35:01.854869 19154 catalog_manager.cc:5495] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc21149b1d9454d924948d17e6cbe06 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "cfc21149b1d9454d924948d17e6cbe06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc21149b1d9454d924948d17e6cbe06" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 } health_report { overall_health: UNKNOWN } } }
I20260926 21:35:01.896330 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.891290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.0.97.2:37179)
0926 21:35:01.891491 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.891496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.891514 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:35:01.891703 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.891707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.891789 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.891911 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:01.892237 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.892916 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.894208 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.894668 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.894759 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.895281 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:01.895749 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:01.895873 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.895970 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.896004 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:35:01.904673 19296 raft_consensus.cc:1261] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 1 FOLLOWER]: Refusing update from remote peer cfc21149b1d9454d924948d17e6cbe06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:01.904673 19557 raft_consensus.cc:1261] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 1 FOLLOWER]: Refusing update from remote peer cfc21149b1d9454d924948d17e6cbe06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:01.905815 19631 consensus_queue.cc:1035] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d591938a2634e049da399b7d7275005" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:01.906172 19626 consensus_queue.cc:1035] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14550af5415642659c5f4365c23e344a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:01.918084 19635 mvcc.cc:205] Tried to move back new op lower bound from 7333718023791132672 to 7333718023581147136. Current Snapshot: MvccSnapshot[applied={T|T < 7333718023791132672}]
I20260926 21:35:01.933029 19155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:35:01.933348 19155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:35:01.937238 19155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0da5103bf034ab38eec55a0bd030a31: Sending DeleteTablet for 3 replicas of tablet d726e72f34db49ffbe24342d3143ac6b
I20260926 21:35:01.938455 19407 tablet_service.cc:1514] Processing DeleteTablet for tablet d726e72f34db49ffbe24342d3143ac6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:01 UTC) from {username='slave'} at 127.0.0.1:41548
I20260926 21:35:01.938465 19276 tablet_service.cc:1514] Processing DeleteTablet for tablet d726e72f34db49ffbe24342d3143ac6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:01 UTC) from {username='slave'} at 127.0.0.1:48232
I20260926 21:35:01.938542 19537 tablet_service.cc:1514] Processing DeleteTablet for tablet d726e72f34db49ffbe24342d3143ac6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:01 UTC) from {username='slave'} at 127.0.0.1:49044
I20260926 21:35:01.940173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 19214
I20260926 21:35:01.950979 19645 tablet_replica.cc:330] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: stopping tablet replica
I20260926 21:35:01.951493 19645 raft_consensus.cc:2229] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:35:01.952216 19645 raft_consensus.cc:2258] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:35:01.952977 19644 tablet_replica.cc:330] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: stopping tablet replica
I20260926 21:35:01.953588 19644 raft_consensus.cc:2229] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:35:01.954178 19644 raft_consensus.cc:2258] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:35:01.956413 19644 ts_tablet_manager.cc:1905] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:01.962026 19645 ts_tablet_manager.cc:1905] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:01.963907 19644 ts_tablet_manager.cc:1918] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:01.964170 19644 log.cc:1196] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/d726e72f34db49ffbe24342d3143ac6b
I20260926 21:35:01.964787 19644 ts_tablet_manager.cc:1939] T d726e72f34db49ffbe24342d3143ac6b P 3d591938a2634e049da399b7d7275005: Deleting consensus metadata
I20260926 21:35:01.966833 19141 catalog_manager.cc:4841] TS 3d591938a2634e049da399b7d7275005 (127.0.97.1:36657): tablet d726e72f34db49ffbe24342d3143ac6b (table test-table [id=364b76d7814e47e78787233c2517b993]) successfully deleted
I20260926 21:35:01.969389 19646 tablet_replica.cc:330] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: stopping tablet replica
I20260926 21:35:01.969867 19646 raft_consensus.cc:2229] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:35:01.970860 19646 raft_consensus.cc:2258] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:35:01.972441 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:01.941768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.0.97.1:36657)
0926 21:35:01.941998 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:01.942007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:01.953907 (+ 11900us) server_negotiation.cc:407] Connection header received
0926 21:35:01.954517 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:01.954527 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:01.954665 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:01.954780 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:01.955507 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.957006 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.966615 (+ 9609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:01.967563 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:01.967701 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:01.970613 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:01.970625 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:01.970643 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:01.971476 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:01.971549 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:01.971576 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:01.971690 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:01.971931 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 21:35:01.972066 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:35:01.973497 19646 ts_tablet_manager.cc:1905] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:01.973697 19645 ts_tablet_manager.cc:1918] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:01.973930 19645 log.cc:1196] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/d726e72f34db49ffbe24342d3143ac6b
I20260926 21:35:01.973953 19336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:01.974870 19645 ts_tablet_manager.cc:1939] T d726e72f34db49ffbe24342d3143ac6b P cfc21149b1d9454d924948d17e6cbe06: Deleting consensus metadata
I20260926 21:35:01.976601 19139 catalog_manager.cc:4841] TS cfc21149b1d9454d924948d17e6cbe06 (127.0.97.2:37179): tablet d726e72f34db49ffbe24342d3143ac6b (table test-table [id=364b76d7814e47e78787233c2517b993]) successfully deleted
I20260926 21:35:01.981285 19646 ts_tablet_manager.cc:1918] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:01.981441 19646 log.cc:1196] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/d726e72f34db49ffbe24342d3143ac6b
I20260926 21:35:01.981909 19646 ts_tablet_manager.cc:1939] T d726e72f34db49ffbe24342d3143ac6b P 14550af5415642659c5f4365c23e344a: Deleting consensus metadata
I20260926 21:35:01.983755 19140 catalog_manager.cc:4841] TS 14550af5415642659c5f4365c23e344a (127.0.97.3:37939): tablet d726e72f34db49ffbe24342d3143ac6b (table test-table [id=364b76d7814e47e78787233c2517b993]) successfully deleted
I20260926 21:35:02.067184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 19214
I20260926 21:35:02.091187 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 19345
I20260926 21:35:02.099584 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:02.092146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.0.97.2:37179)
0926 21:35:02.092314 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:02.092319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:02.092336 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:35:02.092419 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:02.092426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:02.092501 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:02.092589 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:02.093068 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:02.093834 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:02.095900 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:02.096325 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:02.096431 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:02.097704 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:02.097715 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:02.097735 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:02.098610 (+ 875us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:02.098675 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:02.098703 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:02.098816 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:02.099050 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 21:35:02.099160 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:35:02.100435 19467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:02.213611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 19345
I20260926 21:35:02.232048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 19475
I20260926 21:35:02.238387 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:02.232784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.0.97.3:37939)
0926 21:35:02.232949 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:02.232954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:02.233013 (+ 59us) server_negotiation.cc:407] Connection header received
0926 21:35:02.233176 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:02.233180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:02.233262 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:02.233351 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:02.233712 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:02.234486 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:02.235748 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:02.236135 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:02.236227 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:02.237222 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:02.237230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:02.237245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:02.237812 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:02.237857 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:02.237876 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:02.237960 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:02.238065 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 21:35:02.238151 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:35:02.239207 19597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:02.327710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 19475
I20260926 21:35:02.344671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 19124
I20260926 21:35:02.345706 19185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:02.453738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 19124
2026-09-26T21:35:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2795 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:02 dist-test-slave-87bt krb5kdc[19654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:02 dist-test-slave-87bt krb5kdc[19654](info): set up 2 sockets
Sep 26 21:35:02 dist-test-slave-87bt krb5kdc[19654](info): commencing operation
krb5kdc: starting...
W20260926 21:35:03.551024 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:03 dist-test-slave-87bt krb5kdc[19654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:03.801981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38483
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:04.152295 19667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:04.152606 19667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:04.152773 19667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:04.160140 19667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:04.160276 19667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:04.160360 19667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:04.160459 19667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:04.160545 19667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:04.170353 19667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:04.171731 19667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:04.173036 19667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:04.184916 19672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:04.184768 19675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:04.184754 19673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:04.186163 19667 server_base.cc:1034] running on GCE node
I20260926 21:35:04.187067 19667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:04.188616 19667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:04.189895 19667 hybrid_clock.cc:648] HybridClock initialized: now 1790458504189817 us; error 72 us; skew 500 ppm
I20260926 21:35:04.190274 19667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:04 dist-test-slave-87bt krb5kdc[19654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:04.193483 19667 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:04.195182 19667 webserver.cc:460] Webserver started at http://127.0.97.62:39445/ using document root <none> and password file <none>
I20260926 21:35:04.195924 19667 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:04.196087 19667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:04.196411 19667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:04.199594 19667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "884857c190bc4974b4f97c2d1fc6ef2f"
format_stamp: "Formatted at 2026-09-26 21:35:04 on dist-test-slave-87bt"
server_key: "2f2df531c203ef64b27e24cba6768d3c"
server_key_iv: "4f86831a6d6e66abe06ee64808820cb8"
server_key_version: "encryptionkey@0"
I20260926 21:35:04.200414 19667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "884857c190bc4974b4f97c2d1fc6ef2f"
format_stamp: "Formatted at 2026-09-26 21:35:04 on dist-test-slave-87bt"
server_key: "2f2df531c203ef64b27e24cba6768d3c"
server_key_iv: "4f86831a6d6e66abe06ee64808820cb8"
server_key_version: "encryptionkey@0"
I20260926 21:35:04.205983 19667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:35:04.209578 19682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:04.210994 19667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:04.211206 19667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "884857c190bc4974b4f97c2d1fc6ef2f"
format_stamp: "Formatted at 2026-09-26 21:35:04 on dist-test-slave-87bt"
server_key: "2f2df531c203ef64b27e24cba6768d3c"
server_key_iv: "4f86831a6d6e66abe06ee64808820cb8"
server_key_version: "encryptionkey@0"
I20260926 21:35:04.211428 19667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:04.228926 19667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:04.233260 19667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:04.233569 19667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:04.252327 19667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38483
I20260926 21:35:04.252359 19733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38483 every 8 connection(s)
I20260926 21:35:04.253769 19667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:04.256037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 19667
I20260926 21:35:04.256227 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:04.256762 388 external_mini_cluster.cc:1428] Setting key 0507df1be829c54e98540ee18c5ca716
I20260926 21:35:04.258584 19734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:35:04 dist-test-slave-87bt krb5kdc[19654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:04.272466 19734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f: Bootstrap starting.
I20260926 21:35:04.275952 19734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:04.276973 19734 log.cc:824] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:04.279763 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:04.258634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.0.97.62:38483)
0926 21:35:04.259455 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:04.259467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:04.259514 (+ 47us) server_negotiation.cc:407] Connection header received
0926 21:35:04.261113 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:04.261134 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:04.261524 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:04.261862 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:04.263669 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:04.265013 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:04.266879 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:04.267567 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:04.267698 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:04.272501 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:04.272509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:04.272525 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:04.272571 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:04.275635 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:04.276432 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:04.276439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:04.276445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:04.276601 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:04.277000 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:04.277004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:04.277006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:04.277707 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:04.278045 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:04.278354 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 21:35:04.278815 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":359,"threads_started":1}
I20260926 21:35:04.280025 19734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f: No bootstrap required, opened a new log
I20260926 21:35:04.287305 19734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884857c190bc4974b4f97c2d1fc6ef2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } }
I20260926 21:35:04.287703 19734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:04.287813 19734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884857c190bc4974b4f97c2d1fc6ef2f, State: Initialized, Role: FOLLOWER
I20260926 21:35:04.288365 19734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884857c190bc4974b4f97c2d1fc6ef2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } }
I20260926 21:35:04.288607 19734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:04.288749 19734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:04.288928 19734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:04.291136 19734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884857c190bc4974b4f97c2d1fc6ef2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } }
I20260926 21:35:04.291607 19734 leader_election.cc:302] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 884857c190bc4974b4f97c2d1fc6ef2f; no voters:
I20260926 21:35:04.292145 19734 leader_election.cc:288] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:04.292270 19739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:04.293284 19739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [term 1 LEADER]: Becoming Leader. State: Replica: 884857c190bc4974b4f97c2d1fc6ef2f, State: Running, Role: LEADER
I20260926 21:35:04.294104 19739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884857c190bc4974b4f97c2d1fc6ef2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } }
I20260926 21:35:04.294209 19734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:04.297231 19740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "884857c190bc4974b4f97c2d1fc6ef2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884857c190bc4974b4f97c2d1fc6ef2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } } }
I20260926 21:35:04.297688 19740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:04.297245 19741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884857c190bc4974b4f97c2d1fc6ef2f. Latest consensus state: current_term: 1 leader_uuid: "884857c190bc4974b4f97c2d1fc6ef2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884857c190bc4974b4f97c2d1fc6ef2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38483 } } }
I20260926 21:35:04.298379 19741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:04.299135 19745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:04.305823 19745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:04.314985 19745 catalog_manager.cc:1347] Generated new cluster ID: 769a4d2edac44632b364ac6c1e24adea
I20260926 21:35:04.315130 19745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:04.340399 19745 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:04.342322 19745 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:04.356029 19745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f: Generated new TSK 0
I20260926 21:35:04.357129 19745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:35:04.421711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:04.757643 19760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:04.757952 19760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:04.758119 19760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:04.765374 19760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:04.765519 19760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:04.765690 19760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:35:04.775357 19760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:04.776657 19760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:04.777992 19760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:04.791507 19768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:04.791591 19765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:04.791602 19766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:04.792615 19760 server_base.cc:1034] running on GCE node
I20260926 21:35:04.793211 19760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:04.794260 19760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:04.795485 19760 hybrid_clock.cc:648] HybridClock initialized: now 1790458504795431 us; error 44 us; skew 500 ppm
I20260926 21:35:04.795804 19760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:04 dist-test-slave-87bt krb5kdc[19654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:04.799134 19760 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:35:04.800545 19760 webserver.cc:460] Webserver started at http://127.0.97.1:37271/ using document root <none> and password file <none>
I20260926 21:35:04.801245 19760 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:04.801398 19760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:04.801730 19760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:04.804762 19760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "2a9127e1318c46f28fa01d4f510c98c1"
format_stamp: "Formatted at 2026-09-26 21:35:04 on dist-test-slave-87bt"
server_key: "dedd58061596de8ab5235705bade8975"
server_key_iv: "81de28a98f032427abf8c60cdea52c2a"
server_key_version: "encryptionkey@0"
I20260926 21:35:04.805565 19760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2a9127e1318c46f28fa01d4f510c98c1"
format_stamp: "Formatted at 2026-09-26 21:35:04 on dist-test-slave-87bt"
server_key: "dedd58061596de8ab5235705bade8975"
server_key_iv: "81de28a98f032427abf8c60cdea52c2a"
server_key_version: "encryptionkey@0"
I20260926 21:35:04.810746 19760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:35:04.814177 19775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:04.815678 19760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:04.815861 19760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "2a9127e1318c46f28fa01d4f510c98c1"
format_stamp: "Formatted at 2026-09-26 21:35:04 on dist-test-slave-87bt"
server_key: "dedd58061596de8ab5235705bade8975"
server_key_iv: "81de28a98f032427abf8c60cdea52c2a"
server_key_version: "encryptionkey@0"
I20260926 21:35:04.816107 19760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:04.852231 19760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:04.856377 19760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:04.856671 19760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:04.857607 19760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:04.859088 19760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:04.859189 19760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:04.859334 19760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:04.859404 19760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:04.891871 19760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33423
I20260926 21:35:04.891908 19887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33423 every 8 connection(s)
I20260926 21:35:04.893146 19760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:35:04.899278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 19760
I20260926 21:35:04.899502 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:35:04.899989 388 external_mini_cluster.cc:1428] Setting key f4f7722c3fbcf4a09f097d2f90f4a35f
Sep 26 21:35:04 dist-test-slave-87bt krb5kdc[19654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:04.915365 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:04.895849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49779 (local address 127.0.97.62:38483)
0926 21:35:04.896320 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:04.896327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:04.897440 (+ 1113us) server_negotiation.cc:407] Connection header received
0926 21:35:04.899409 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:04.899416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:04.899525 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:04.899656 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:04.901421 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:04.902448 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:04.904085 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:04.904797 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:04.904938 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:04.908849 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:04.908858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:04.908866 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:04.908914 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:04.911825 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:04.912842 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:04.912846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:04.912849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:04.912968 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:04.913483 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:04.913486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:04.913488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:04.913742 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:04.913850 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:04.914853 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0926 21:35:04.915078 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260926 21:35:04.916883 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:04.896530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38483 (local address 127.0.97.1:49779)
0926 21:35:04.897262 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:04.897302 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:04.899105 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:04.899895 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:04.899904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:04.900547 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:04.901243 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:04.901262 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:04.902616 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:04.902620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:04.903929 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:04.903941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:04.905029 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:04.905039 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:04.905253 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:04.905983 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:04.906011 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:04.908600 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:04.911987 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:04.912000 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:04.912022 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:04.912718 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:04.913086 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:04.913094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:04.913100 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:04.913373 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:04.913875 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:04.913887 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:04.914439 (+ 552us) client_negotiation.cc:769] Sending connection context
0926 21:35:04.914773 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 21:35:04.915383 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":313,"threads_started":1}
I20260926 21:35:04.918538 19888 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38483
I20260926 21:35:04.918859 19888 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:04.919775 19888 heartbeater.cc:511] Master 127.0.97.62:38483 requested a full tablet report, sending...
I20260926 21:35:04.921932 19699 ts_manager.cc:194] Registered new tserver with Master: 2a9127e1318c46f28fa01d4f510c98c1 (127.0.97.1:33423)
I20260926 21:35:04.923524 19699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49779
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:35:04.955586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:05.281718 19894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:05.282047 19894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:05.282217 19894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:05.289501 19894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:05.289655 19894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:05.289839 19894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:35:05.299885 19894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:05.301265 19894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:05.302676 19894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:05.315907 19902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:05.315951 19899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:05.315981 19900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:05.316743 19894 server_base.cc:1034] running on GCE node
I20260926 21:35:05.317301 19894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:05.318253 19894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:05.319550 19894 hybrid_clock.cc:648] HybridClock initialized: now 1790458505319485 us; error 40 us; skew 500 ppm
I20260926 21:35:05.319895 19894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:05 dist-test-slave-87bt krb5kdc[19654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:05.323137 19894 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:35:05.324633 19894 webserver.cc:460] Webserver started at http://127.0.97.2:42151/ using document root <none> and password file <none>
I20260926 21:35:05.325377 19894 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:05.325517 19894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:05.325840 19894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:05.328862 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "cae6b53bf03e4b45a7497d4b739a9771"
format_stamp: "Formatted at 2026-09-26 21:35:05 on dist-test-slave-87bt"
server_key: "bef4cca400762c5d4703e10b47548a55"
server_key_iv: "3ce481c47092a06993344c553d3e1e84"
server_key_version: "encryptionkey@0"
I20260926 21:35:05.329691 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cae6b53bf03e4b45a7497d4b739a9771"
format_stamp: "Formatted at 2026-09-26 21:35:05 on dist-test-slave-87bt"
server_key: "bef4cca400762c5d4703e10b47548a55"
server_key_iv: "3ce481c47092a06993344c553d3e1e84"
server_key_version: "encryptionkey@0"
I20260926 21:35:05.335244 19894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:35:05.339151 19909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:05.340585 19894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:35:05.340773 19894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "cae6b53bf03e4b45a7497d4b739a9771"
format_stamp: "Formatted at 2026-09-26 21:35:05 on dist-test-slave-87bt"
server_key: "bef4cca400762c5d4703e10b47548a55"
server_key_iv: "3ce481c47092a06993344c553d3e1e84"
server_key_version: "encryptionkey@0"
I20260926 21:35:05.340987 19894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:05.360198 19894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:05.364198 19894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:05.364500 19894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:05.365556 19894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:05.366936 19894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:05.367009 19894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:05.367172 19894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:05.367221 19894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:05.399622 19894 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36251
I20260926 21:35:05.399729 20021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36251 every 8 connection(s)
I20260926 21:35:05.400903 19894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:35:05.410125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 19894
I20260926 21:35:05.410318 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:35:05.410771 388 external_mini_cluster.cc:1428] Setting key 94dee68e2a5c06776d29cb216d7ea07f
Sep 26 21:35:05 dist-test-slave-87bt krb5kdc[19654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:05.423691 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.403543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44795 (local address 127.0.97.62:38483)
0926 21:35:05.403726 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:05.403730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:05.404885 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 21:35:05.407201 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:05.407206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:05.407308 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:05.407422 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:05.409055 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.409875 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.411744 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.412229 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.412331 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:05.416357 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:05.416367 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:05.416375 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:05.416423 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:05.420304 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:05.421211 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:05.421220 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:05.421226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:05.421397 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:05.421911 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:05.421914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:05.421916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:05.422205 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:05.422315 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:05.423257 (+ 942us) server_negotiation.cc:299] Negotiation successful
0926 21:35:05.423443 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:35:05.425330 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.404042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38483 (local address 127.0.97.2:44795)
0926 21:35:05.404724 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:05.404760 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:05.406887 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:05.407691 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:05.407701 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:05.408249 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:05.408857 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:05.408874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.410091 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.410095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:05.411599 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:05.411613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.412509 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.412516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:05.412830 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:05.413551 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:05.413579 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:05.416075 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:05.420476 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:05.420483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:05.420499 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:05.421066 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:05.421514 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:05.421521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:05.421525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:05.421793 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:05.422338 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:05.422348 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:05.422860 (+ 512us) client_negotiation.cc:769] Sending connection context
0926 21:35:05.423178 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 21:35:05.423776 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":285,"threads_started":1}
I20260926 21:35:05.426954 20022 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38483
I20260926 21:35:05.427278 20022 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:05.428056 20022 heartbeater.cc:511] Master 127.0.97.62:38483 requested a full tablet report, sending...
I20260926 21:35:05.429605 19699 ts_manager.cc:194] Registered new tserver with Master: cae6b53bf03e4b45a7497d4b739a9771 (127.0.97.2:36251)
I20260926 21:35:05.430522 19699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44795
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:35:05.467609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:05.797843 20027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:05.798127 20027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:05.798312 20027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:05.805835 20027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:05.805956 20027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:05.806147 20027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:35:05.816252 20027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:05.817637 20027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:05.819026 20027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:05.833014 20032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:05.832948 20035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:05.832978 20033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:05.833920 20027 server_base.cc:1034] running on GCE node
I20260926 21:35:05.834672 20027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:05.835695 20027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:05.836942 20027 hybrid_clock.cc:648] HybridClock initialized: now 1790458505836896 us; error 51 us; skew 500 ppm
I20260926 21:35:05.837281 20027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:05 dist-test-slave-87bt krb5kdc[19654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:05.840447 20027 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:35:05.841888 20027 webserver.cc:460] Webserver started at http://127.0.97.3:43767/ using document root <none> and password file <none>
I20260926 21:35:05.842784 20027 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:05.842929 20027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:05.843257 20027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:05.846308 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "802a2267643445e38d507edebfc89b8d"
format_stamp: "Formatted at 2026-09-26 21:35:05 on dist-test-slave-87bt"
server_key: "750abe17e49fe66b4e3705c32d294a1d"
server_key_iv: "c4858d42ca5a044918a9838c4ae7f05d"
server_key_version: "encryptionkey@0"
I20260926 21:35:05.847168 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "802a2267643445e38d507edebfc89b8d"
format_stamp: "Formatted at 2026-09-26 21:35:05 on dist-test-slave-87bt"
server_key: "750abe17e49fe66b4e3705c32d294a1d"
server_key_iv: "c4858d42ca5a044918a9838c4ae7f05d"
server_key_version: "encryptionkey@0"
I20260926 21:35:05.852883 20027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:35:05.856716 20042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:05.858078 20027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:35:05.858253 20027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "802a2267643445e38d507edebfc89b8d"
format_stamp: "Formatted at 2026-09-26 21:35:05 on dist-test-slave-87bt"
server_key: "750abe17e49fe66b4e3705c32d294a1d"
server_key_iv: "c4858d42ca5a044918a9838c4ae7f05d"
server_key_version: "encryptionkey@0"
I20260926 21:35:05.858495 20027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:05.871367 20027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:05.875336 20027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:05.875639 20027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:05.876592 20027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:05.877889 20027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:05.877997 20027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:05.878146 20027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:05.878221 20027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:05.911499 20027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35567
I20260926 21:35:05.911542 20154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35567 every 8 connection(s)
I20260926 21:35:05.912756 20027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:35:05.922106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 20027
I20260926 21:35:05.922392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:35:05.922845 388 external_mini_cluster.cc:1428] Setting key 5f20943dceb5cc41641d2fe907036037
I20260926 21:35:05.926640 19888 heartbeater.cc:503] Master 127.0.97.62:38483 was elected leader, sending a full tablet report...
Sep 26 21:35:05 dist-test-slave-87bt krb5kdc[19654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:05.934407 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.915272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59111 (local address 127.0.97.62:38483)
0926 21:35:05.915521 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:05.915527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:05.916716 (+ 1189us) server_negotiation.cc:407] Connection header received
0926 21:35:05.918586 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:05.918591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:05.918663 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:05.918743 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:05.920521 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.921295 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.923215 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.923775 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.923925 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:05.928101 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:05.928108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:05.928115 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:05.928165 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:05.931073 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:05.932203 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:05.932209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:05.932213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:05.932383 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:05.932870 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:05.932874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:05.932876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:05.933169 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:05.933290 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:05.933978 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 21:35:05.934156 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:35:05.935478 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.915821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38483 (local address 127.0.97.3:59111)
0926 21:35:05.916569 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:05.916606 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:05.918238 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:05.919029 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:05.919039 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:05.919666 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:05.920314 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:05.920343 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.921453 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.921457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:05.923040 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:05.923060 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.924038 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.924049 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:05.924323 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:05.925054 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:05.925084 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:05.927862 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:05.931272 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:05.931285 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:05.931304 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:05.932064 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:05.932529 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:05.932533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:05.932536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:05.932729 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:05.933283 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:05.933289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:05.933625 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 21:35:05.933853 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 21:35:05.934277 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":336,"threads_started":1}
I20260926 21:35:05.937130 20155 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38483
I20260926 21:35:05.937543 20155 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:05.938391 20155 heartbeater.cc:511] Master 127.0.97.62:38483 requested a full tablet report, sending...
I20260926 21:35:05.939842 19698 ts_manager.cc:194] Registered new tserver with Master: 802a2267643445e38d507edebfc89b8d (127.0.97.3:35567)
I20260926 21:35:05.940733 19698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59111
I20260926 21:35:05.950760 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:35:05.967402 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.954717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.0.97.62:38483)
0926 21:35:05.954873 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:05.954878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:05.955310 (+ 432us) server_negotiation.cc:407] Connection header received
0926 21:35:05.955543 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:05.955547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:05.955608 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:05.955713 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:05.957040 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.957751 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.959200 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.959599 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.959729 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:05.962225 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:05.962230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:05.962233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:05.962266 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:05.964669 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:05.965355 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:05.965365 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:05.965367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:05.965486 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:05.965850 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:05.965855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:05.965858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:05.966194 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:05.966497 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:05.966858 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 21:35:05.967089 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:35:05.971455 19698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:35:05.979610 19698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:35:06.003214 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.992540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33423 (local address 127.0.0.1:36528)
0926 21:35:05.993809 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:05.993846 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:05.994068 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:05.995840 (+ 1772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:05.995848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:05.995883 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:05.996346 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:05.996357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.998197 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.998203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.000384 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:06.000396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.002068 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.002074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.002265 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.002355 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 21:35:06.002521 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 21:35:06.002672 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":3,"thread_start_us":2595,"threads_started":1}
I20260926 21:35:06.003836 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.992567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.0.97.1:33423)
0926 21:35:05.995162 (+ 2595us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:05.995168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:05.995191 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:35:05.995314 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:05.995322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:05.995608 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:05.996502 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:05.996666 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.998041 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.000536 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.001814 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.002026 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.002264 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.003340 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.003530 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2370,"thread_start_us":2197,"threads_started":1}
I20260926 21:35:06.003971 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.992862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36251 (local address 127.0.0.1:58504)
0926 21:35:05.995407 (+ 2545us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:05.995428 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:05.995633 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:05.996569 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:05.996576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:05.996611 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:05.997042 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:05.997056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.998935 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:05.998956 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.001092 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:06.001106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.002986 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.002995 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.003369 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.003410 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 21:35:06.003502 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 21:35:06.003623 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2400,"thread_start_us":781,"threads_started":1}
I20260926 21:35:06.004593 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.993452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.0.97.2:36251)
0926 21:35:05.994629 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:05.994634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:05.995520 (+ 886us) server_negotiation.cc:407] Connection header received
0926 21:35:05.995835 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:05.995843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:05.996085 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:05.996261 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:05.997240 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:05.998817 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.001250 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.002778 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.002912 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.003116 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.004212 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.004365 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":912,"threads_started":1}
I20260926 21:35:06.007500 19823 tablet_service.cc:1467] Processing CreateTablet for tablet 349b2860ac3249a087e788b88f651af0 (DEFAULT_TABLE table=test-table [id=6910602a04674348a1faa3cf7eaee082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:06.007967 19957 tablet_service.cc:1467] Processing CreateTablet for tablet 349b2860ac3249a087e788b88f651af0 (DEFAULT_TABLE table=test-table [id=6910602a04674348a1faa3cf7eaee082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:06.008674 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.996230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35567 (local address 127.0.0.1:38238)
0926 21:35:05.997593 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:05.997615 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:05.997737 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:05.998762 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:05.998771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:05.998799 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:05.999478 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:05.999490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.002278 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.002284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.006018 (+ 3734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:06.006035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.007851 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.007857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.008036 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.008068 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:35:06.008156 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 21:35:06.008291 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":29,"thread_start_us":1111,"threads_started":1}
I20260926 21:35:06.009089 19823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349b2860ac3249a087e788b88f651af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:06.009328 19957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349b2860ac3249a087e788b88f651af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:06.009727 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:05.995826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.0.97.3:35567)
0926 21:35:05.998291 (+ 2465us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:05.998297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:05.998323 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:35:05.998389 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:05.998394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:05.998599 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:05.998770 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:06.000639 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.002109 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.006227 (+ 4118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.007637 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.007775 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.009153 (+ 1378us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.009234 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.009413 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2333,"thread_start_us":2183,"threads_started":1}
I20260926 21:35:06.013410 20090 tablet_service.cc:1467] Processing CreateTablet for tablet 349b2860ac3249a087e788b88f651af0 (DEFAULT_TABLE table=test-table [id=6910602a04674348a1faa3cf7eaee082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:06.015020 20090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349b2860ac3249a087e788b88f651af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:06.020779 20173 tablet_bootstrap.cc:492] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: Bootstrap starting.
I20260926 21:35:06.023659 20174 tablet_bootstrap.cc:492] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: Bootstrap starting.
I20260926 21:35:06.025575 20173 tablet_bootstrap.cc:654] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:06.026940 20173 log.cc:824] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:06.027303 20174 tablet_bootstrap.cc:654] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:06.028425 20174 log.cc:824] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:06.029172 20176 tablet_bootstrap.cc:492] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: Bootstrap starting.
I20260926 21:35:06.030335 20173 tablet_bootstrap.cc:492] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: No bootstrap required, opened a new log
I20260926 21:35:06.030702 20173 ts_tablet_manager.cc:1397] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 21:35:06.032306 20176 tablet_bootstrap.cc:654] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:06.033012 20174 tablet_bootstrap.cc:492] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: No bootstrap required, opened a new log
I20260926 21:35:06.033273 20174 ts_tablet_manager.cc:1397] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 21:35:06.033285 20176 log.cc:824] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:06.035629 20176 tablet_bootstrap.cc:492] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: No bootstrap required, opened a new log
I20260926 21:35:06.035874 20176 ts_tablet_manager.cc:1397] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 21:35:06.041271 20173 raft_consensus.cc:348] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.041811 20173 raft_consensus.cc:374] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:06.041929 20173 raft_consensus.cc:729] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae6b53bf03e4b45a7497d4b739a9771, State: Initialized, Role: FOLLOWER
I20260926 21:35:06.042685 20173 consensus_queue.cc:260] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.043094 20176 raft_consensus.cc:348] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.043589 20176 raft_consensus.cc:374] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:06.043689 20176 raft_consensus.cc:729] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802a2267643445e38d507edebfc89b8d, State: Initialized, Role: FOLLOWER
I20260926 21:35:06.043948 20022 heartbeater.cc:503] Master 127.0.97.62:38483 was elected leader, sending a full tablet report...
I20260926 21:35:06.044319 20176 consensus_queue.cc:260] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.044240 20174 raft_consensus.cc:348] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.044641 20174 raft_consensus.cc:374] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:06.044745 20174 raft_consensus.cc:729] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9127e1318c46f28fa01d4f510c98c1, State: Initialized, Role: FOLLOWER
I20260926 21:35:06.045497 20155 heartbeater.cc:503] Master 127.0.97.62:38483 was elected leader, sending a full tablet report...
I20260926 21:35:06.046224 20176 ts_tablet_manager.cc:1428] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260926 21:35:06.046380 20173 ts_tablet_manager.cc:1428] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260926 21:35:06.045401 20174 consensus_queue.cc:260] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.049717 20174 ts_tablet_manager.cc:1428] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 21:35:06.055635 20180 raft_consensus.cc:482] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:35:06.055902 20180 raft_consensus.cc:504] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.057608 20180 leader_election.cc:288] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a9127e1318c46f28fa01d4f510c98c1 (127.0.97.1:33423), cae6b53bf03e4b45a7497d4b739a9771 (127.0.97.2:36251)
I20260926 21:35:06.064574 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:06.058151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41931 (local address 127.0.97.2:36251)
0926 21:35:06.058352 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:06.058359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:06.058378 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:35:06.058614 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:06.058622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:06.058716 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:06.058828 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:06.059266 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.060262 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.062559 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.063724 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.063870 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.063979 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.064173 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.064315 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:35:06.064563 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:06.057898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36251 (local address 127.0.97.3:41931)
0926 21:35:06.058118 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:06.058139 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:06.058260 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:06.058876 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:06.058881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:06.058916 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:06.059139 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:06.059149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.060394 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.060398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.062316 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:06.062332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.063875 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.063879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.064018 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.064043 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 21:35:06.064119 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 21:35:06.064215 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260926 21:35:06.065155 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:06.058342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59091 (local address 127.0.97.1:33423)
0926 21:35:06.058594 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:06.058603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:06.059117 (+ 514us) server_negotiation.cc:407] Connection header received
0926 21:35:06.059274 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:06.059279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:06.059360 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:06.059454 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:06.059906 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.060773 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.062966 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.064229 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.064358 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.064455 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.064825 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.064977 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:35:06.065760 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:06.057925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33423 (local address 127.0.97.3:59091)
0926 21:35:06.059006 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:06.059023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:06.059144 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:06.059476 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:06.059479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:06.059498 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:06.059763 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:06.059770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.060910 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.060915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.062814 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:06.062824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.064442 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.064448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:06.064624 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.064663 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 21:35:06.065344 (+ 681us) client_negotiation.cc:240] Negotiation successful
0926 21:35:06.065504 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":56,"thread_start_us":653,"threads_started":1}
I20260926 21:35:06.065689 19977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349b2860ac3249a087e788b88f651af0" candidate_uuid: "802a2267643445e38d507edebfc89b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae6b53bf03e4b45a7497d4b739a9771" is_pre_election: true
I20260926 21:35:06.066263 19977 raft_consensus.cc:2454] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802a2267643445e38d507edebfc89b8d in term 0.
I20260926 21:35:06.066890 19843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349b2860ac3249a087e788b88f651af0" candidate_uuid: "802a2267643445e38d507edebfc89b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9127e1318c46f28fa01d4f510c98c1" is_pre_election: true
I20260926 21:35:06.067065 20046 leader_election.cc:302] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 802a2267643445e38d507edebfc89b8d, cae6b53bf03e4b45a7497d4b739a9771; no voters:
I20260926 21:35:06.067389 19843 raft_consensus.cc:2454] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802a2267643445e38d507edebfc89b8d in term 0.
I20260926 21:35:06.067495 20180 raft_consensus.cc:2781] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:35:06.067667 20180 raft_consensus.cc:482] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:35:06.067783 20180 raft_consensus.cc:3037] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:06.070012 20180 raft_consensus.cc:504] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.070700 20180 leader_election.cc:288] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [CANDIDATE]: Term 1 election: Requested vote from peers 2a9127e1318c46f28fa01d4f510c98c1 (127.0.97.1:33423), cae6b53bf03e4b45a7497d4b739a9771 (127.0.97.2:36251)
I20260926 21:35:06.071163 19843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349b2860ac3249a087e788b88f651af0" candidate_uuid: "802a2267643445e38d507edebfc89b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9127e1318c46f28fa01d4f510c98c1"
I20260926 21:35:06.071245 19977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349b2860ac3249a087e788b88f651af0" candidate_uuid: "802a2267643445e38d507edebfc89b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae6b53bf03e4b45a7497d4b739a9771"
I20260926 21:35:06.071379 19843 raft_consensus.cc:3037] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:06.071439 19977 raft_consensus.cc:3037] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:06.074632 19977 raft_consensus.cc:2454] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802a2267643445e38d507edebfc89b8d in term 1.
I20260926 21:35:06.074632 19843 raft_consensus.cc:2454] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802a2267643445e38d507edebfc89b8d in term 1.
I20260926 21:35:06.075032 20044 leader_election.cc:302] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9127e1318c46f28fa01d4f510c98c1, 802a2267643445e38d507edebfc89b8d; no voters:
I20260926 21:35:06.075425 20180 raft_consensus.cc:2781] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:06.075932 20180 raft_consensus.cc:686] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 1 LEADER]: Becoming Leader. State: Replica: 802a2267643445e38d507edebfc89b8d, State: Running, Role: LEADER
I20260926 21:35:06.076493 20180 consensus_queue.cc:237] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } }
I20260926 21:35:06.081532 19697 catalog_manager.cc:5495] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 802a2267643445e38d507edebfc89b8d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "802a2267643445e38d507edebfc89b8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "802a2267643445e38d507edebfc89b8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 } health_report { overall_health: UNKNOWN } } }
I20260926 21:35:06.102547 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:06.097303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.0.97.3:35567)
0926 21:35:06.097480 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:06.097485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:06.097518 (+ 33us) server_negotiation.cc:407] Connection header received
0926 21:35:06.097703 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:06.097707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:06.097793 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:06.097920 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:06.098306 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.099075 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.100406 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.100792 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.100951 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.101470 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:06.102000 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:06.102163 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.102217 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.102292 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:35:06.110239 19977 raft_consensus.cc:1261] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 1 FOLLOWER]: Refusing update from remote peer 802a2267643445e38d507edebfc89b8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:06.110244 19843 raft_consensus.cc:1261] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 1 FOLLOWER]: Refusing update from remote peer 802a2267643445e38d507edebfc89b8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:06.111227 20183 consensus_queue.cc:1035] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae6b53bf03e4b45a7497d4b739a9771" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:06.111558 20180 consensus_queue.cc:1035] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9127e1318c46f28fa01d4f510c98c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:06.143404 19697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:35:06.143685 19697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:35:06.147305 19697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 884857c190bc4974b4f97c2d1fc6ef2f: Sending DeleteTablet for 3 replicas of tablet 349b2860ac3249a087e788b88f651af0
I20260926 21:35:06.148610 19957 tablet_service.cc:1514] Processing DeleteTablet for tablet 349b2860ac3249a087e788b88f651af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58504
I20260926 21:35:06.148626 20090 tablet_service.cc:1514] Processing DeleteTablet for tablet 349b2860ac3249a087e788b88f651af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38238
I20260926 21:35:06.149367 19823 tablet_service.cc:1514] Processing DeleteTablet for tablet 349b2860ac3249a087e788b88f651af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36528
I20260926 21:35:06.149901 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 19760
I20260926 21:35:06.150030 20196 tablet_replica.cc:330] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: stopping tablet replica
I20260926 21:35:06.150523 20196 raft_consensus.cc:2229] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:35:06.151245 20196 raft_consensus.cc:2258] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:35:06.152096 20197 tablet_replica.cc:330] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: stopping tablet replica
I20260926 21:35:06.152583 20197 raft_consensus.cc:2229] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:35:06.153195 20197 raft_consensus.cc:2258] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:35:06.153981 20196 ts_tablet_manager.cc:1905] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:06.157215 20197 ts_tablet_manager.cc:1905] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:06.163048 20196 ts_tablet_manager.cc:1918] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:06.163254 20196 log.cc:1196] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/349b2860ac3249a087e788b88f651af0
I20260926 21:35:06.163786 20196 ts_tablet_manager.cc:1939] T 349b2860ac3249a087e788b88f651af0 P 802a2267643445e38d507edebfc89b8d: Deleting consensus metadata
I20260926 21:35:06.165632 19684 catalog_manager.cc:4841] TS 802a2267643445e38d507edebfc89b8d (127.0.97.3:35567): tablet 349b2860ac3249a087e788b88f651af0 (table test-table [id=6910602a04674348a1faa3cf7eaee082]) successfully deleted
I20260926 21:35:06.170284 20198 tablet_replica.cc:330] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: stopping tablet replica
I20260926 21:35:06.170722 20198 raft_consensus.cc:2229] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:35:06.171201 20198 raft_consensus.cc:2258] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 21:35:06 dist-test-slave-87bt krb5kdc[19654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:35:06.173246 20198 ts_tablet_manager.cc:1905] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:06.174690 20197 ts_tablet_manager.cc:1918] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:06.175223 20197 log.cc:1196] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/349b2860ac3249a087e788b88f651af0
I20260926 21:35:06.175797 20197 ts_tablet_manager.cc:1939] T 349b2860ac3249a087e788b88f651af0 P cae6b53bf03e4b45a7497d4b739a9771: Deleting consensus metadata
I20260926 21:35:06.177975 19686 catalog_manager.cc:4841] TS cae6b53bf03e4b45a7497d4b739a9771 (127.0.97.2:36251): tablet 349b2860ac3249a087e788b88f651af0 (table test-table [id=6910602a04674348a1faa3cf7eaee082]) successfully deleted
I20260926 21:35:06.180439 20198 ts_tablet_manager.cc:1918] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:06.180564 20198 log.cc:1196] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/349b2860ac3249a087e788b88f651af0
I20260926 21:35:06.181069 20198 ts_tablet_manager.cc:1939] T 349b2860ac3249a087e788b88f651af0 P 2a9127e1318c46f28fa01d4f510c98c1: Deleting consensus metadata
I20260926 21:35:06.182612 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:06.152355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.0.97.1:33423)
0926 21:35:06.152530 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:06.152536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:06.161871 (+ 9335us) server_negotiation.cc:407] Connection header received
0926 21:35:06.162064 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:06.162070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:06.162187 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:06.162298 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:06.164260 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.165417 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.167407 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.168337 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.168510 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.173865 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:06.173882 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:06.173899 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:06.173959 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:06.178287 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:06.179448 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:06.179458 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:06.179462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:06.179615 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:06.180073 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:06.180078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:06.180082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:06.180698 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:06.181030 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.181960 (+ 930us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.182207 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:35:06.182925 19684 catalog_manager.cc:4841] TS 2a9127e1318c46f28fa01d4f510c98c1 (127.0.97.1:33423): tablet 349b2860ac3249a087e788b88f651af0 (table test-table [id=6910602a04674348a1faa3cf7eaee082]) successfully deleted
I20260926 21:35:06.183334 19883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:06.275429 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 19760
I20260926 21:35:06.292893 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 19894
Sep 26 21:35:06 dist-test-slave-87bt krb5kdc[19654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:35:06.310058 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:06.293768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.0.97.2:36251)
0926 21:35:06.293986 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:06.293992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:06.294015 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:35:06.294085 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:06.294091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:06.294190 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:06.294310 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:06.295755 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.296891 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.298180 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.298865 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.298996 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.303328 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:06.303348 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:06.303369 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:06.303414 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:06.307255 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:06.308009 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:06.308019 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:06.308023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:06.308181 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:06.308538 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:06.308543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:06.308546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:06.309033 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:06.309331 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.309490 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.309709 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:35:06.310777 20017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:06.400044 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 19894
I20260926 21:35:06.418093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20027
Sep 26 21:35:06 dist-test-slave-87bt krb5kdc[19654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:35:06.437002 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:06.419327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.0.97.3:35567)
0926 21:35:06.419551 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:06.419558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:06.419580 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:35:06.419646 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:06.419653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:06.419745 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:06.419869 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:06.421162 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.422388 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.423770 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:06.424470 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:06.424628 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:06.429352 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:06.429370 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:06.429390 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:06.429436 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:06.433773 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:06.434680 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:06.434691 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:06.434694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:06.434912 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:06.435316 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:06.435322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:06.435325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:06.435821 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:06.436174 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:06.436345 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 21:35:06.436572 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:35:06.437950 20150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:06.560484 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 20027
I20260926 21:35:06.580085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 19667
I20260926 21:35:06.581264 19729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:06.692193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 19667
2026-09-26T21:35:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4238 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T21:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:06Z Disabled control of system clock
I20260926 21:35:06.767957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45121
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:35:07.106470 20207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:07.106792 20207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:07.106876 20207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:07.114120 20207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:07.114233 20207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:07.114271 20207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:07.114320 20207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:07.114357 20207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:07.124679 20207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:07.126048 20207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:07.128304 20207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:07.140872 20212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:07.140852 20213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:07.141116 20215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:07.141726 20207 server_base.cc:1034] running on GCE node
I20260926 21:35:07.142633 20207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:07.144115 20207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:07.145350 20207 hybrid_clock.cc:648] HybridClock initialized: now 1790458507145286 us; error 62 us; skew 500 ppm
I20260926 21:35:07.145715 20207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:35:07.148204 20207 webserver.cc:460] Webserver started at http://127.0.97.62:40641/ using document root <none> and password file <none>
I20260926 21:35:07.148916 20207 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:07.149051 20207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:07.149379 20207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:07.152411 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "23df63c0eccb4ca9a099a6b2afcf8406"
format_stamp: "Formatted at 2026-09-26 21:35:07 on dist-test-slave-87bt"
server_key: "17e5312c1fa32a93de164820169b7a53"
server_key_iv: "d242836bc6ae93dc21a078c30555b4b6"
server_key_version: "encryptionkey@0"
I20260926 21:35:07.153232 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "23df63c0eccb4ca9a099a6b2afcf8406"
format_stamp: "Formatted at 2026-09-26 21:35:07 on dist-test-slave-87bt"
server_key: "17e5312c1fa32a93de164820169b7a53"
server_key_iv: "d242836bc6ae93dc21a078c30555b4b6"
server_key_version: "encryptionkey@0"
I20260926 21:35:07.158349 20207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:35:07.161788 20221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:07.163214 20207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 21:35:07.163444 20207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "23df63c0eccb4ca9a099a6b2afcf8406"
format_stamp: "Formatted at 2026-09-26 21:35:07 on dist-test-slave-87bt"
server_key: "17e5312c1fa32a93de164820169b7a53"
server_key_iv: "d242836bc6ae93dc21a078c30555b4b6"
server_key_version: "encryptionkey@0"
I20260926 21:35:07.163666 20207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:07.185010 20207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:07.185871 20207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:07.186177 20207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:07.205603 20207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45121
I20260926 21:35:07.205631 20272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45121 every 8 connection(s)
I20260926 21:35:07.207176 20207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:07.211193 20273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:07.211850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 20207
I20260926 21:35:07.212021 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:07.212608 388 external_mini_cluster.cc:1428] Setting key 3dcf1b06358900b9f43c620a3cb15079
I20260926 21:35:07.225318 20273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406: Bootstrap starting.
I20260926 21:35:07.227196 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:07.214645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.0.97.62:45121)
0926 21:35:07.215709 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:07.215720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:07.215768 (+ 48us) server_negotiation.cc:407] Connection header received
0926 21:35:07.217539 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:07.217564 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:07.218107 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:07.218539 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:07.219086 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:07.220371 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:07.222400 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:07.223245 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:07.223394 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:07.224683 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:07.224691 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:07.224708 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:07.225266 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:07.225352 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:07.225374 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:07.225537 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:07.225818 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 21:35:07.226146 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":407,"threads_started":1}
I20260926 21:35:07.228749 20273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:07.229787 20273 log.cc:824] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:07.232849 20273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406: No bootstrap required, opened a new log
I20260926 21:35:07.240311 20273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df63c0eccb4ca9a099a6b2afcf8406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45121 } }
I20260926 21:35:07.240685 20273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:07.240759 20273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23df63c0eccb4ca9a099a6b2afcf8406, State: Initialized, Role: FOLLOWER
I20260926 21:35:07.241330 20273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df63c0eccb4ca9a099a6b2afcf8406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45121 } }
I20260926 21:35:07.241536 20273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:07.241628 20273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:07.241742 20273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:07.243968 20273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df63c0eccb4ca9a099a6b2afcf8406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45121 } }
I20260926 21:35:07.244565 20273 leader_election.cc:302] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23df63c0eccb4ca9a099a6b2afcf8406; no voters:
I20260926 21:35:07.245246 20273 leader_election.cc:288] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:07.245374 20278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:07.246052 20278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [term 1 LEADER]: Becoming Leader. State: Replica: 23df63c0eccb4ca9a099a6b2afcf8406, State: Running, Role: LEADER
I20260926 21:35:07.246713 20278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df63c0eccb4ca9a099a6b2afcf8406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45121 } }
I20260926 21:35:07.247566 20273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:07.249578 20279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23df63c0eccb4ca9a099a6b2afcf8406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df63c0eccb4ca9a099a6b2afcf8406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45121 } } }
I20260926 21:35:07.249997 20279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:07.249737 20280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23df63c0eccb4ca9a099a6b2afcf8406. Latest consensus state: current_term: 1 leader_uuid: "23df63c0eccb4ca9a099a6b2afcf8406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df63c0eccb4ca9a099a6b2afcf8406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45121 } } }
I20260926 21:35:07.250164 20280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:07.252159 20285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:07.258939 20285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:07.268307 20285 catalog_manager.cc:1347] Generated new cluster ID: 7cdae83543b24538a61d428ce542aa01
I20260926 21:35:07.268502 20285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:07.283339 20285 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:07.285122 20285 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:07.298770 20285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406: Generated new TSK 0
I20260926 21:35:07.299806 20285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:07.310498 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45121
--builtin_ntp_servers=127.0.97.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:35:07.652588 20297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:07.652884 20297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:07.653029 20297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:07.660378 20297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:07.660485 20297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:07.660590 20297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:35:07.670143 20297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:07.671525 20297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:07.673188 20297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:07.686352 20303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:07.686373 20305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:07.686434 20302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:07.687080 20297 server_base.cc:1034] running on GCE node
I20260926 21:35:07.687705 20297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:07.688628 20297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:07.689831 20297 hybrid_clock.cc:648] HybridClock initialized: now 1790458507689795 us; error 31 us; skew 500 ppm
I20260926 21:35:07.690115 20297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:35:07.692343 20297 webserver.cc:460] Webserver started at http://127.0.97.1:42423/ using document root <none> and password file <none>
I20260926 21:35:07.692982 20297 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:07.693117 20297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:07.693436 20297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:07.696244 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/data/instance:
uuid: "bc778782188a47e6921c3bc07a0e9b7e"
format_stamp: "Formatted at 2026-09-26 21:35:07 on dist-test-slave-87bt"
server_key: "941fdae759c0c2b6fd0fbc9029324c13"
server_key_iv: "f000b923d55fb517ba10bb021a35c369"
server_key_version: "encryptionkey@0"
I20260926 21:35:07.696972 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bc778782188a47e6921c3bc07a0e9b7e"
format_stamp: "Formatted at 2026-09-26 21:35:07 on dist-test-slave-87bt"
server_key: "941fdae759c0c2b6fd0fbc9029324c13"
server_key_iv: "f000b923d55fb517ba10bb021a35c369"
server_key_version: "encryptionkey@0"
I20260926 21:35:07.701793 20297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 21:35:07.705096 20311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:07.706362 20297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:35:07.706559 20297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/wal
uuid: "bc778782188a47e6921c3bc07a0e9b7e"
format_stamp: "Formatted at 2026-09-26 21:35:07 on dist-test-slave-87bt"
server_key: "941fdae759c0c2b6fd0fbc9029324c13"
server_key_iv: "f000b923d55fb517ba10bb021a35c369"
server_key_version: "encryptionkey@0"
I20260926 21:35:07.706745 20297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:07.750278 20297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:07.751199 20297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:07.751446 20297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:07.752523 20297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:07.754186 20297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:07.754294 20297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:07.754452 20297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:07.754549 20297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 21:35:07.786125 20297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34579
I20260926 21:35:07.786170 20423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34579 every 8 connection(s)
I20260926 21:35:07.787465 20297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:35:07.795085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 20297
I20260926 21:35:07.795248 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:35:07.795663 388 external_mini_cluster.cc:1428] Setting key be35f0cd73eae89cd72596ba03186639
I20260926 21:35:07.798610 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45121
--builtin_ntp_servers=127.0.97.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:35:07.800603 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:07.789974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52621 (local address 127.0.97.62:45121)
0926 21:35:07.790473 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:07.790480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:07.791543 (+ 1063us) server_negotiation.cc:407] Connection header received
0926 21:35:07.793378 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:07.793384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:07.793482 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:07.793606 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:07.794793 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:07.795657 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:07.797380 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:07.797984 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:07.798113 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:07.799611 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:07.799619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:07.799628 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:07.799654 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:07.799716 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:07.799734 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:07.799834 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:07.800056 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 21:35:07.800171 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":231,"threads_started":1}
I20260926 21:35:07.801561 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:07.790478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45121 (local address 127.0.97.1:52621)
0926 21:35:07.791379 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:07.791416 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:07.793081 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:07.793863 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:07.793872 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:07.793970 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:07.794616 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:07.794635 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:07.795805 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:07.795812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:07.797204 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:07.797243 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:07.798191 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:07.798196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:07.798403 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:07.799233 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:35:07.799237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:07.799307 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:35:07.799410 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:07.799841 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:07.799849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:07.799854 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:35:07.800003 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 21:35:07.800302 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260926 21:35:07.803180 20424 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45121
I20260926 21:35:07.803507 20424 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:07.804232 20424 heartbeater.cc:511] Master 127.0.97.62:45121 requested a full tablet report, sending...
I20260926 21:35:07.806281 20238 ts_manager.cc:194] Registered new tserver with Master: bc778782188a47e6921c3bc07a0e9b7e (127.0.97.1:34579)
I20260926 21:35:07.807771 20238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52621
W20260926 21:35:08.125598 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:08.125908 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:08.126081 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:08.133517 20428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:08.133622 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:08.133805 20428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:35:08.143435 20428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:08.144711 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:08.146473 20428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:08.159390 20433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:08.159476 20434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:08.159437 20436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:08.160653 20428 server_base.cc:1034] running on GCE node
I20260926 21:35:08.161188 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:08.162043 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:08.163276 20428 hybrid_clock.cc:648] HybridClock initialized: now 1790458508163223 us; error 56 us; skew 500 ppm
I20260926 21:35:08.163609 20428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:35:08.165942 20428 webserver.cc:460] Webserver started at http://127.0.97.2:38571/ using document root <none> and password file <none>
I20260926 21:35:08.166716 20428 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:08.166841 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:08.167155 20428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:08.170058 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/data/instance:
uuid: "54c71954a77c48b1ae7f8bda5b36453b"
format_stamp: "Formatted at 2026-09-26 21:35:08 on dist-test-slave-87bt"
server_key: "5e1ca035e1f75210636cb17d4514e512"
server_key_iv: "84824154ebd0b33124f3667b29cf2199"
server_key_version: "encryptionkey@0"
I20260926 21:35:08.170846 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/wal/instance:
uuid: "54c71954a77c48b1ae7f8bda5b36453b"
format_stamp: "Formatted at 2026-09-26 21:35:08 on dist-test-slave-87bt"
server_key: "5e1ca035e1f75210636cb17d4514e512"
server_key_iv: "84824154ebd0b33124f3667b29cf2199"
server_key_version: "encryptionkey@0"
I20260926 21:35:08.175900 20428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:35:08.179346 20442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:08.180720 20428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:35:08.180842 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/wal
uuid: "54c71954a77c48b1ae7f8bda5b36453b"
format_stamp: "Formatted at 2026-09-26 21:35:08 on dist-test-slave-87bt"
server_key: "5e1ca035e1f75210636cb17d4514e512"
server_key_iv: "84824154ebd0b33124f3667b29cf2199"
server_key_version: "encryptionkey@0"
I20260926 21:35:08.181021 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:08.198721 20428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:08.199553 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:08.199798 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:08.200727 20428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:08.202204 20428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:08.202291 20428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:08.202364 20428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:08.202451 20428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:08.238109 20428 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40183
I20260926 21:35:08.238197 20554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40183 every 8 connection(s)
I20260926 21:35:08.239480 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:35:08.244191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 20428
I20260926 21:35:08.244364 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:35:08.244917 388 external_mini_cluster.cc:1428] Setting key 74368a1fcbdd783a49469b576f3ecf38
I20260926 21:35:08.248215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45121
--builtin_ntp_servers=127.0.97.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:35:08.252993 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.242142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45965 (local address 127.0.97.62:45121)
0926 21:35:08.242367 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.242373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.243732 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 21:35:08.246099 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.246103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.246166 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.246252 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:08.247645 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.248386 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.249982 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.250376 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.250567 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.252007 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:08.252015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:08.252023 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:08.252049 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:08.252093 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:08.252113 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:08.252210 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.252451 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.252589 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 21:35:08.253999 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.242683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45121 (local address 127.0.97.2:45965)
0926 21:35:08.243575 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:08.243610 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:08.245805 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:08.246588 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:08.246599 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:08.246720 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:08.247472 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.247497 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.248571 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.248576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.249823 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.249835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.250667 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.250674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.250915 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.251666 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:35:08.251669 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:08.251709 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:35:08.251792 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:08.252213 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:08.252220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:08.252225 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:35:08.252407 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 21:35:08.252791 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":258,"threads_started":1}
I20260926 21:35:08.255498 20555 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45121
I20260926 21:35:08.255795 20555 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:08.256536 20555 heartbeater.cc:511] Master 127.0.97.62:45121 requested a full tablet report, sending...
I20260926 21:35:08.257968 20238 ts_manager.cc:194] Registered new tserver with Master: 54c71954a77c48b1ae7f8bda5b36453b (127.0.97.2:40183)
I20260926 21:35:08.259212 20238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45965
W20260926 21:35:08.586809 20558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:08.587085 20558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:08.587170 20558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:08.594487 20558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:08.594590 20558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:08.594694 20558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:35:08.603775 20558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:08.605057 20558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:08.606935 20558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:08.620477 20563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:08.620415 20564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:08.620638 20566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:08.621085 20558 server_base.cc:1034] running on GCE node
I20260926 21:35:08.621786 20558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:08.622889 20558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:08.624120 20558 hybrid_clock.cc:648] HybridClock initialized: now 1790458508624064 us; error 40 us; skew 500 ppm
I20260926 21:35:08.624516 20558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:35:08.627286 20558 webserver.cc:460] Webserver started at http://127.0.97.3:41377/ using document root <none> and password file <none>
I20260926 21:35:08.628284 20558 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:08.628419 20558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:08.628710 20558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:08.631810 20558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/data/instance:
uuid: "dcf18ef4a4044916aad6b83b73d1f656"
format_stamp: "Formatted at 2026-09-26 21:35:08 on dist-test-slave-87bt"
server_key: "bef5d69d83fb3cddac7034e64cec0e0c"
server_key_iv: "6fdd04a4d087fc75b3230a7b49588e78"
server_key_version: "encryptionkey@0"
I20260926 21:35:08.632635 20558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dcf18ef4a4044916aad6b83b73d1f656"
format_stamp: "Formatted at 2026-09-26 21:35:08 on dist-test-slave-87bt"
server_key: "bef5d69d83fb3cddac7034e64cec0e0c"
server_key_iv: "6fdd04a4d087fc75b3230a7b49588e78"
server_key_version: "encryptionkey@0"
I20260926 21:35:08.638332 20558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:35:08.642030 20572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:08.643566 20558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 21:35:08.643743 20558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/wal
uuid: "dcf18ef4a4044916aad6b83b73d1f656"
format_stamp: "Formatted at 2026-09-26 21:35:08 on dist-test-slave-87bt"
server_key: "bef5d69d83fb3cddac7034e64cec0e0c"
server_key_iv: "6fdd04a4d087fc75b3230a7b49588e78"
server_key_version: "encryptionkey@0"
I20260926 21:35:08.643980 20558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:08.665432 20558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:08.666299 20558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:08.666625 20558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:08.667670 20558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:35:08.669257 20558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:35:08.669368 20558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:08.669504 20558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:35:08.669600 20558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:08.703780 20558 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40053
I20260926 21:35:08.703879 20684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40053 every 8 connection(s)
I20260926 21:35:08.705142 20558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:35:08.706169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 20558
I20260926 21:35:08.706341 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:35:08.706969 388 external_mini_cluster.cc:1428] Setting key 94dffcb7a9d116f7865a1ecc66c62426
I20260926 21:35:08.718403 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.708201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60947 (local address 127.0.97.62:45121)
0926 21:35:08.708412 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.708419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.710398 (+ 1979us) server_negotiation.cc:407] Connection header received
0926 21:35:08.712284 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.712288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.712359 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.712464 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:08.713509 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.714235 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.715894 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.716291 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.716394 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.717742 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:08.717747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:08.717750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:08.717768 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:08.717798 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:08.717810 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:08.717879 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.718082 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.718162 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:35:08.719559 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.709019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45121 (local address 127.0.97.3:60947)
0926 21:35:08.710206 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:08.710275 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:08.712015 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:08.712667 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:08.712674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:08.712764 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:08.713361 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.713376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.714395 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.714400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.715754 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.715767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.716508 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.716513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.716717 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.717420 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:35:08.717423 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:08.717463 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:35:08.717554 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:08.717895 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:08.717902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:08.717908 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:35:08.718055 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 21:35:08.718392 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":335,"threads_started":1}
I20260926 21:35:08.721299 20685 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45121
I20260926 21:35:08.721657 20685 heartbeater.cc:464] Registering TS with master...
I20260926 21:35:08.722555 20685 heartbeater.cc:511] Master 127.0.97.62:45121 requested a full tablet report, sending...
I20260926 21:35:08.723978 20238 ts_manager.cc:194] Registered new tserver with Master: dcf18ef4a4044916aad6b83b73d1f656 (127.0.97.3:40053)
I20260926 21:35:08.724830 20238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60947
I20260926 21:35:08.735759 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:35:08.746084 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.739991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.0.97.62:45121)
0926 21:35:08.740202 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.740210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.740546 (+ 336us) server_negotiation.cc:407] Connection header received
0926 21:35:08.740765 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.740769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.740847 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.740960 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:08.741466 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.742166 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.743662 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.744116 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.744264 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.745359 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:08.745364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:08.745367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:08.745386 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:08.745424 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:08.745437 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:08.745533 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.745674 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.745772 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:35:08.750306 20238 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:35:08.757668 20238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:35:08.779853 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.769718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.0.97.2:40183)
0926 21:35:08.770793 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.770799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.770850 (+ 51us) server_negotiation.cc:407] Connection header received
0926 21:35:08.770960 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.770967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.771256 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.771461 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:08.772876 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.774483 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.777319 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.778702 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.778855 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.779120 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.779345 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.779509 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":832,"threads_started":1}
I20260926 21:35:08.779852 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.769617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40183 (local address 127.0.0.1:47482)
0926 21:35:08.770533 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:08.770579 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:08.770728 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:08.771737 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:08.771750 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:08.771787 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:08.772184 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.772197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.774657 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.774666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.777077 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.777090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.778887 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.778893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.779096 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.779151 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 21:35:08.779301 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 21:35:08.779499 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":521,"threads_started":1}
I20260926 21:35:08.780645 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.770921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34579 (local address 127.0.0.1:42766)
0926 21:35:08.771486 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:08.771505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:08.771613 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:08.773474 (+ 1861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:08.773479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:08.773506 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:08.773841 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.773850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.775916 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.775927 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.778086 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.778100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.779960 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.779966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.780144 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.780178 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:35:08.780243 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 21:35:08.780363 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":267,"threads_started":1}
I20260926 21:35:08.781670 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.771223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.0.97.1:34579)
0926 21:35:08.772682 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.772691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.772718 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:35:08.772951 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.772963 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.773263 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.774019 (+ 756us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:08.774232 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.775724 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.778302 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.779727 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.780880 (+ 1153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.781118 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.781185 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.781344 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":1208,"threads_started":1}
I20260926 21:35:08.783406 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.771550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40053 (local address 127.0.0.1:60794)
0926 21:35:08.772090 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:08.772114 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:08.772239 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:08.775615 (+ 3376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:08.775620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:08.775645 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:08.776030 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.776045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.777763 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.777770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.779727 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.779741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.782724 (+ 2983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.782731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.782904 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.782938 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:35:08.783018 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 21:35:08.783133 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":759,"threads_started":1}
I20260926 21:35:08.784395 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.772136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.0.97.3:40053)
0926 21:35:08.775077 (+ 2941us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.775085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.775116 (+ 31us) server_negotiation.cc:407] Connection header received
0926 21:35:08.775204 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.775212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.775426 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.775594 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:08.776281 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.777636 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.779951 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.781339 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.783585 (+ 2246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.783832 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.783901 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.784083 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2774,"thread_start_us":2627,"threads_started":1}
I20260926 21:35:08.783932 20490 tablet_service.cc:1467] Processing CreateTablet for tablet b5ed9ba56c594c508ac3f852d28171c9 (DEFAULT_TABLE table=test-table [id=4a9df42287194ce087e35ef8fee04d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:08.784665 20359 tablet_service.cc:1467] Processing CreateTablet for tablet b5ed9ba56c594c508ac3f852d28171c9 (DEFAULT_TABLE table=test-table [id=4a9df42287194ce087e35ef8fee04d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:08.785326 20490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ed9ba56c594c508ac3f852d28171c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:08.785979 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ed9ba56c594c508ac3f852d28171c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:08.793156 20620 tablet_service.cc:1467] Processing CreateTablet for tablet b5ed9ba56c594c508ac3f852d28171c9 (DEFAULT_TABLE table=test-table [id=4a9df42287194ce087e35ef8fee04d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:35:08.794520 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ed9ba56c594c508ac3f852d28171c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:08.799171 20703 tablet_bootstrap.cc:492] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Bootstrap starting.
I20260926 21:35:08.802181 20703 tablet_bootstrap.cc:654] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:08.803082 20704 tablet_bootstrap.cc:492] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: Bootstrap starting.
I20260926 21:35:08.803509 20703 log.cc:824] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:08.806744 20704 tablet_bootstrap.cc:654] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:08.806846 20703 tablet_bootstrap.cc:492] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: No bootstrap required, opened a new log
I20260926 21:35:08.807118 20703 ts_tablet_manager.cc:1397] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 21:35:08.807973 20704 log.cc:824] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:08.810348 20707 tablet_bootstrap.cc:492] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Bootstrap starting.
I20260926 21:35:08.810791 20424 heartbeater.cc:503] Master 127.0.97.62:45121 was elected leader, sending a full tablet report...
I20260926 21:35:08.811192 20704 tablet_bootstrap.cc:492] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: No bootstrap required, opened a new log
I20260926 21:35:08.811455 20704 ts_tablet_manager.cc:1397] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260926 21:35:08.814232 20707 tablet_bootstrap.cc:654] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:08.815543 20707 log.cc:824] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:08.819234 20703 raft_consensus.cc:348] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.819720 20703 raft_consensus.cc:374] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:08.819819 20703 raft_consensus.cc:729] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c71954a77c48b1ae7f8bda5b36453b, State: Initialized, Role: FOLLOWER
I20260926 21:35:08.820505 20703 consensus_queue.cc:260] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.821898 20555 heartbeater.cc:503] Master 127.0.97.62:45121 was elected leader, sending a full tablet report...
I20260926 21:35:08.822595 20703 ts_tablet_manager.cc:1428] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260926 21:35:08.823992 20704 raft_consensus.cc:348] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.824492 20704 raft_consensus.cc:374] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:08.824592 20704 raft_consensus.cc:729] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc778782188a47e6921c3bc07a0e9b7e, State: Initialized, Role: FOLLOWER
I20260926 21:35:08.825345 20704 consensus_queue.cc:260] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.828173 20704 ts_tablet_manager.cc:1428] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: Time spent starting tablet: real 0.017s user 0.011s sys 0.002s
I20260926 21:35:08.829432 20707 tablet_bootstrap.cc:492] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: No bootstrap required, opened a new log
I20260926 21:35:08.829730 20707 ts_tablet_manager.cc:1397] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.013s
I20260926 21:35:08.839674 20707 raft_consensus.cc:348] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.840179 20707 raft_consensus.cc:374] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:08.840263 20707 raft_consensus.cc:729] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf18ef4a4044916aad6b83b73d1f656, State: Initialized, Role: FOLLOWER
I20260926 21:35:08.840960 20707 consensus_queue.cc:260] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.842207 20685 heartbeater.cc:503] Master 127.0.97.62:45121 was elected leader, sending a full tablet report...
I20260926 21:35:08.842967 20707 ts_tablet_manager.cc:1428] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 21:35:08.903540 20710 raft_consensus.cc:482] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:35:08.903815 20710 raft_consensus.cc:504] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.905530 20710 leader_election.cc:288] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54c71954a77c48b1ae7f8bda5b36453b (127.0.97.2:40183), dcf18ef4a4044916aad6b83b73d1f656 (127.0.97.3:40053)
I20260926 21:35:08.912077 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.905894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43161 (local address 127.0.97.2:40183)
0926 21:35:08.906146 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.906154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.906578 (+ 424us) server_negotiation.cc:407] Connection header received
0926 21:35:08.906853 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.906862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.906971 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.907074 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:08.907650 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.908333 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.910227 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.911204 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.911329 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.911426 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.911698 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.911817 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:35:08.912184 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.905693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40183 (local address 127.0.97.1:43161)
0926 21:35:08.906449 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:08.906471 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:08.906677 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:08.907114 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:08.907121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:08.907157 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:08.907502 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.907510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.908462 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.908469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.910049 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.910061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.911381 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.911395 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.911544 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.911567 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:35:08.911672 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 21:35:08.911797 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":48,"thread_start_us":338,"threads_started":1}
I20260926 21:35:08.912886 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.906387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40053 (local address 127.0.97.1:57543)
0926 21:35:08.907312 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:08.907327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:08.907475 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:08.907823 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:08.907828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:08.907851 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:35:08.908155 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.908163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.909150 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.909155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.910864 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:08.910876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.912063 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.912069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:08.912274 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.912303 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 21:35:08.912392 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 21:35:08.912530 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":18,"thread_start_us":475,"threads_started":1}
I20260926 21:35:08.912998 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.906731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57543 (local address 127.0.97.3:40053)
0926 21:35:08.906966 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.906973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.907416 (+ 443us) server_negotiation.cc:407] Connection header received
0926 21:35:08.907629 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.907638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.907698 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.907807 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:35:08.908304 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.909001 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.911095 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.911869 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.912018 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.912118 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.912702 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.912800 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 21:35:08.913223 20510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ed9ba56c594c508ac3f852d28171c9" candidate_uuid: "bc778782188a47e6921c3bc07a0e9b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c71954a77c48b1ae7f8bda5b36453b" is_pre_election: true
I20260926 21:35:08.913820 20510 raft_consensus.cc:2454] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc778782188a47e6921c3bc07a0e9b7e in term 0.
I20260926 21:35:08.913923 20640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ed9ba56c594c508ac3f852d28171c9" candidate_uuid: "bc778782188a47e6921c3bc07a0e9b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf18ef4a4044916aad6b83b73d1f656" is_pre_election: true
I20260926 21:35:08.914413 20640 raft_consensus.cc:2454] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc778782188a47e6921c3bc07a0e9b7e in term 0.
I20260926 21:35:08.914572 20312 leader_election.cc:302] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c71954a77c48b1ae7f8bda5b36453b, bc778782188a47e6921c3bc07a0e9b7e; no voters:
I20260926 21:35:08.915010 20710 raft_consensus.cc:2781] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:35:08.915164 20710 raft_consensus.cc:482] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:35:08.915270 20710 raft_consensus.cc:3037] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:08.917441 20710 raft_consensus.cc:504] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.918051 20710 leader_election.cc:288] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [CANDIDATE]: Term 1 election: Requested vote from peers 54c71954a77c48b1ae7f8bda5b36453b (127.0.97.2:40183), dcf18ef4a4044916aad6b83b73d1f656 (127.0.97.3:40053)
I20260926 21:35:08.918625 20640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ed9ba56c594c508ac3f852d28171c9" candidate_uuid: "bc778782188a47e6921c3bc07a0e9b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf18ef4a4044916aad6b83b73d1f656"
I20260926 21:35:08.918592 20510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ed9ba56c594c508ac3f852d28171c9" candidate_uuid: "bc778782188a47e6921c3bc07a0e9b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c71954a77c48b1ae7f8bda5b36453b"
I20260926 21:35:08.918836 20640 raft_consensus.cc:3037] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:08.918852 20510 raft_consensus.cc:3037] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:08.921195 20640 raft_consensus.cc:2454] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc778782188a47e6921c3bc07a0e9b7e in term 1.
I20260926 21:35:08.921195 20510 raft_consensus.cc:2454] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc778782188a47e6921c3bc07a0e9b7e in term 1.
I20260926 21:35:08.921728 20312 leader_election.cc:302] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c71954a77c48b1ae7f8bda5b36453b, bc778782188a47e6921c3bc07a0e9b7e; no voters:
I20260926 21:35:08.922042 20710 raft_consensus.cc:2781] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:08.922574 20710 raft_consensus.cc:686] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 1 LEADER]: Becoming Leader. State: Replica: bc778782188a47e6921c3bc07a0e9b7e, State: Running, Role: LEADER
I20260926 21:35:08.923084 20710 consensus_queue.cc:237] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } }
I20260926 21:35:08.928224 20237 catalog_manager.cc:5495] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e reported cstate change: term changed from 0 to 1, leader changed from <none> to bc778782188a47e6921c3bc07a0e9b7e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bc778782188a47e6921c3bc07a0e9b7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc778782188a47e6921c3bc07a0e9b7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 } health_report { overall_health: UNKNOWN } } }
W20260926 21:35:08.956875 20686 tablet.cc:2369] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:35:08.979338 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:08.974055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.0.97.1:34579)
0926 21:35:08.974265 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:08.974273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:08.974293 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:35:08.974400 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:08.974404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:08.974567 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:08.974718 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:08.975077 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.975782 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.977154 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:08.977590 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:08.977721 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:08.978194 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:08.978773 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:08.978944 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:08.978994 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 21:35:08.979072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:35:08.987800 20640 raft_consensus.cc:1261] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 1 FOLLOWER]: Refusing update from remote peer bc778782188a47e6921c3bc07a0e9b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:08.987798 20510 raft_consensus.cc:1261] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 1 FOLLOWER]: Refusing update from remote peer bc778782188a47e6921c3bc07a0e9b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:08.988754 20714 consensus_queue.cc:1035] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcf18ef4a4044916aad6b83b73d1f656" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:08.989162 20710 consensus_queue.cc:1035] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c71954a77c48b1ae7f8bda5b36453b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:35:08.991520 20556 tablet.cc:2369] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:35:09.000578 20717 mvcc.cc:205] Tried to move back new op lower bound from 7333718052803641344 to 7333718052553089024. Current Snapshot: MvccSnapshot[applied={T|T < 7333718052803641344}]
I20260926 21:35:09.012926 20238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:35:09.013226 20238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:35:09.016829 20238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23df63c0eccb4ca9a099a6b2afcf8406: Sending DeleteTablet for 3 replicas of tablet b5ed9ba56c594c508ac3f852d28171c9
I20260926 21:35:09.018168 20620 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ed9ba56c594c508ac3f852d28171c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:09 UTC) from {username='slave'} at 127.0.0.1:60794
I20260926 21:35:09.018175 20490 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ed9ba56c594c508ac3f852d28171c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:09 UTC) from {username='slave'} at 127.0.0.1:47482
I20260926 21:35:09.018379 20359 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ed9ba56c594c508ac3f852d28171c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:35:09 UTC) from {username='slave'} at 127.0.0.1:42766
I20260926 21:35:09.019249 20727 tablet_replica.cc:330] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: stopping tablet replica
I20260926 21:35:09.019805 20728 tablet_replica.cc:330] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: stopping tablet replica
I20260926 21:35:09.019977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20297
I20260926 21:35:09.020380 20728 raft_consensus.cc:2229] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:35:09.020993 20728 raft_consensus.cc:2258] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:35:09.021040 20727 raft_consensus.cc:2229] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:35:09.021857 20727 raft_consensus.cc:2258] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:35:09.023938 20728 ts_tablet_manager.cc:1905] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:09.026286 20727 ts_tablet_manager.cc:1905] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:09.033200 20728 ts_tablet_manager.cc:1918] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:09.033438 20727 ts_tablet_manager.cc:1918] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:09.033507 20728 log.cc:1196] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-2/wal/wals/b5ed9ba56c594c508ac3f852d28171c9
I20260926 21:35:09.033613 20727 log.cc:1196] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-1/wal/wals/b5ed9ba56c594c508ac3f852d28171c9
I20260926 21:35:09.034160 20727 ts_tablet_manager.cc:1939] T b5ed9ba56c594c508ac3f852d28171c9 P 54c71954a77c48b1ae7f8bda5b36453b: Deleting consensus metadata
I20260926 21:35:09.034299 20728 ts_tablet_manager.cc:1939] T b5ed9ba56c594c508ac3f852d28171c9 P dcf18ef4a4044916aad6b83b73d1f656: Deleting consensus metadata
I20260926 21:35:09.036271 20222 catalog_manager.cc:4841] TS 54c71954a77c48b1ae7f8bda5b36453b (127.0.97.2:40183): tablet b5ed9ba56c594c508ac3f852d28171c9 (table test-table [id=4a9df42287194ce087e35ef8fee04d46]) successfully deleted
I20260926 21:35:09.036653 20222 catalog_manager.cc:4841] TS dcf18ef4a4044916aad6b83b73d1f656 (127.0.97.3:40053): tablet b5ed9ba56c594c508ac3f852d28171c9 (table test-table [id=4a9df42287194ce087e35ef8fee04d46]) successfully deleted
I20260926 21:35:09.037350 20729 tablet_replica.cc:330] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: stopping tablet replica
I20260926 21:35:09.037767 20729 raft_consensus.cc:2229] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:35:09.038347 20729 raft_consensus.cc:2258] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:35:09.039978 20729 ts_tablet_manager.cc:1905] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:35:09.045496 20729 ts_tablet_manager.cc:1918] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:35:09.045688 20729 log.cc:1196] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: Deleting WAL directory at /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458187148965-388-0/minicluster-data/ts-0/wal/wals/b5ed9ba56c594c508ac3f852d28171c9
I20260926 21:35:09.046309 20729 ts_tablet_manager.cc:1939] T b5ed9ba56c594c508ac3f852d28171c9 P bc778782188a47e6921c3bc07a0e9b7e: Deleting consensus metadata
I20260926 21:35:09.046334 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:09.021785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.97.1:34579)
0926 21:35:09.021950 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:09.021957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:09.031024 (+ 9067us) server_negotiation.cc:407] Connection header received
0926 21:35:09.031240 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:09.031246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:09.031367 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:09.031490 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:09.032179 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:09.033393 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:09.035449 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:09.036076 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:09.036184 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:09.044372 (+ 8188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:09.044400 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:09.044419 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:09.045242 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:09.045316 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:09.045357 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:09.045528 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:09.045741 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 21:35:09.045900 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:35:09.047607 20419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:09.146287 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 20297
I20260926 21:35:09.146375 20222 catalog_manager.cc:4841] TS bc778782188a47e6921c3bc07a0e9b7e (127.0.97.1:34579): tablet b5ed9ba56c594c508ac3f852d28171c9 (table test-table [id=4a9df42287194ce087e35ef8fee04d46]) successfully deleted
I20260926 21:35:09.165494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20428
I20260926 21:35:09.172271 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:09.166343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.0.97.2:40183)
0926 21:35:09.166576 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:09.166581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:09.166597 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:35:09.166751 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:09.166756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:09.166864 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:09.166952 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:09.167338 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:09.168196 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:09.169463 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:09.169835 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:09.169927 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:09.170960 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:09.170968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:09.170982 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:09.171575 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:09.171619 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:09.171639 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:09.171721 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:09.171829 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 21:35:09.171914 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260926 21:35:09.172940 20550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:09.276111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 20428
I20260926 21:35:09.293215 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20558
I20260926 21:35:09.300633 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:09.294005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.0.97.3:40053)
0926 21:35:09.294191 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:09.294198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:09.294242 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:35:09.294540 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:09.294548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:09.294645 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:09.294762 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:09.295168 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:09.295909 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:09.297227 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:09.297639 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:09.297779 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:09.299095 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:09.299107 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:09.299122 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:09.299748 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:09.299802 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:09.299825 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:09.299976 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:09.300138 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 21:35:09.300315 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:35:09.301373 20680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:09.390537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 20558
I20260926 21:35:09.407049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20207
I20260926 21:35:09.408104 20268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:09.515138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 20207
2026-09-26T21:35:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2822 ms)
[----------] 6 tests from EncryptionPolicyTest (21581 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:09 dist-test-slave-87bt krb5kdc[20737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:09 dist-test-slave-87bt krb5kdc[20737](info): set up 2 sockets
Sep 26 21:35:09 dist-test-slave-87bt krb5kdc[20737](info): commencing operation
krb5kdc: starting...
W20260926 21:35:10.630815 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:10 dist-test-slave-87bt krb5kdc[20737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:10.889822 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:11.220453 20750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:11.220853 20750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:11.221050 20750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:11.228675 20750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:11.228878 20750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:11.228994 20750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:11.229106 20750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:11.229182 20750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:11.240260 20750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:11.241684 20750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:11.243070 20750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:11.255275 20755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:11.255388 20756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:11.255856 20758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:11.256150 20750 server_base.cc:1034] running on GCE node
I20260926 21:35:11.257012 20750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:11.258543 20750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:11.259771 20750 hybrid_clock.cc:648] HybridClock initialized: now 1790458511259717 us; error 57 us; skew 500 ppm
I20260926 21:35:11.260185 20750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:11 dist-test-slave-87bt krb5kdc[20737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:11.263405 20750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:11.264974 20750 webserver.cc:460] Webserver started at http://127.0.0.1:35319/ using document root <none> and password file <none>
I20260926 21:35:11.265699 20750 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:11.265856 20750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:11.266168 20750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:11.269480 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "c2e2616b884a4fd4a24eb63a260ca3fa"
format_stamp: "Formatted at 2026-09-26 21:35:11 on dist-test-slave-87bt"
server_key: "bc726cc9f75dfd29bce248534d9d55bc"
server_key_iv: "e4e9eb79f9f79266bc78d727d14ada56"
server_key_version: "encryptionkey@0"
I20260926 21:35:11.270303 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "c2e2616b884a4fd4a24eb63a260ca3fa"
format_stamp: "Formatted at 2026-09-26 21:35:11 on dist-test-slave-87bt"
server_key: "bc726cc9f75dfd29bce248534d9d55bc"
server_key_iv: "e4e9eb79f9f79266bc78d727d14ada56"
server_key_version: "encryptionkey@0"
I20260926 21:35:11.275712 20750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:35:11.279403 20765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:11.280855 20750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:35:11.281039 20750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "c2e2616b884a4fd4a24eb63a260ca3fa"
format_stamp: "Formatted at 2026-09-26 21:35:11 on dist-test-slave-87bt"
server_key: "bc726cc9f75dfd29bce248534d9d55bc"
server_key_iv: "e4e9eb79f9f79266bc78d727d14ada56"
server_key_version: "encryptionkey@0"
I20260926 21:35:11.281240 20750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:11.296569 20750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:11.300750 20750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:11.301070 20750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:11.319976 20750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42379
I20260926 21:35:11.320008 20816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42379 every 8 connection(s)
I20260926 21:35:11.321503 20750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:11.324935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 20750
I20260926 21:35:11.325238 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:11.325522 20817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:11.326056 388 external_mini_cluster.cc:1428] Setting key 965846e3dd77d70396c8627967b77f96
I20260926 21:35:11.339813 20817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa: Bootstrap starting.
Sep 26 21:35:11 dist-test-slave-87bt krb5kdc[20737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:35:11.343775 20817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:11.344898 20817 log.cc:824] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:11.348023 20817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa: No bootstrap required, opened a new log
I20260926 21:35:11.352633 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:11.328264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41207 (local address 127.0.0.1:42379)
0926 21:35:11.329180 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:11.329194 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:11.329247 (+ 53us) server_negotiation.cc:407] Connection header received
0926 21:35:11.331659 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:11.331688 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:11.332170 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:11.332541 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:11.334298 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:11.335696 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:11.337697 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:11.338691 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:11.338866 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:11.343919 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:11.343931 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:11.343953 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:11.344014 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:11.347392 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:11.348318 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:11.348327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:11.348338 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:11.348577 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:11.349070 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:11.349075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:11.349078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:11.349780 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:11.350134 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:11.350726 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 21:35:11.351142 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":463,"threads_started":1}
I20260926 21:35:11.357278 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:11.357972 20817 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e2616b884a4fd4a24eb63a260ca3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42379 } }
I20260926 21:35:11.358419 20817 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:11.358566 20817 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e2616b884a4fd4a24eb63a260ca3fa, State: Initialized, Role: FOLLOWER
I20260926 21:35:11.359180 20817 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e2616b884a4fd4a24eb63a260ca3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42379 } }
I20260926 21:35:11.359427 20817 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:11.359565 20817 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:11.359753 20817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:11.361991 20817 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e2616b884a4fd4a24eb63a260ca3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42379 } }
I20260926 21:35:11.362582 20817 leader_election.cc:302] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2e2616b884a4fd4a24eb63a260ca3fa; no voters:
I20260926 21:35:11.363322 20817 leader_election.cc:288] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:11.363422 20826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:11.366001 20817 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:11.367229 20826 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [term 1 LEADER]: Becoming Leader. State: Replica: c2e2616b884a4fd4a24eb63a260ca3fa, State: Running, Role: LEADER
I20260926 21:35:11.367918 20826 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e2616b884a4fd4a24eb63a260ca3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42379 } }
I20260926 21:35:11.372283 20828 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2e2616b884a4fd4a24eb63a260ca3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e2616b884a4fd4a24eb63a260ca3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42379 } } }
I20260926 21:35:11.372746 20828 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:11.373795 20829 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2e2616b884a4fd4a24eb63a260ca3fa. Latest consensus state: current_term: 1 leader_uuid: "c2e2616b884a4fd4a24eb63a260ca3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e2616b884a4fd4a24eb63a260ca3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42379 } } }
I20260926 21:35:11.374212 20829 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:11.375597 20832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:11.381872 20832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:11.391160 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:11.362952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37271 (local address 127.0.0.1:42379)
0926 21:35:11.363180 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:11.363190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:11.370588 (+ 7398us) server_negotiation.cc:407] Connection header received
0926 21:35:11.370844 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:11.370850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:11.370976 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:11.371105 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:11.373462 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:11.374769 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:11.376807 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:11.377556 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:11.377725 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:11.382156 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:11.382166 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:11.382175 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:11.382230 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:11.386862 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:11.387930 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:11.387937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:11.387941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:11.388090 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:11.388691 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:11.388697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:11.388701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:11.389163 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:11.389423 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:11.390562 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0926 21:35:11.390747 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:35:11.397485 20832 catalog_manager.cc:1347] Generated new cluster ID: d3227b5b8a88465abadd8f50ace73085
I20260926 21:35:11.397651 20832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:11.427045 20832 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:11.428596 20832 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:11.436712 20832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2e2616b884a4fd4a24eb63a260ca3fa: Generated new TSK 0
I20260926 21:35:11.437641 20832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:11.449560 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20750
I20260926 21:35:11.451059 20812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:11.545552 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 20750
W20260926 21:35:11.560199 20735 connection.cc:535] client connection to 127.0.0.1:42379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42379): Connection reset by peer (error 104)
2026-09-26T21:35:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:11 dist-test-slave-87bt krb5kdc[20849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:11 dist-test-slave-87bt krb5kdc[20849](info): set up 2 sockets
Sep 26 21:35:11 dist-test-slave-87bt krb5kdc[20849](info): commencing operation
krb5kdc: starting...
W20260926 21:35:12.650781 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:12 dist-test-slave-87bt krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:12.905706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:13.235028 20862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:13.235334 20862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:13.235427 20862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:13.242712 20862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:13.242841 20862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:13.242888 20862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:13.242954 20862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:13.242998 20862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:13.253039 20862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:13.254318 20862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:13.255672 20862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:13.267768 20870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:13.267868 20867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:13.267799 20868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:13.268720 20862 server_base.cc:1034] running on GCE node
I20260926 21:35:13.269497 20862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:13.270975 20862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:13.272192 20862 hybrid_clock.cc:648] HybridClock initialized: now 1790458513272140 us; error 40 us; skew 500 ppm
I20260926 21:35:13.272570 20862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:13 dist-test-slave-87bt krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:13.275753 20862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:13.277202 20862 webserver.cc:460] Webserver started at http://127.0.0.1:45023/ using document root <none> and password file <none>
I20260926 21:35:13.277841 20862 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:13.277940 20862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:13.278185 20862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:13.281100 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "03de1f5a91284c87beaee9c3558f62da"
format_stamp: "Formatted at 2026-09-26 21:35:13 on dist-test-slave-87bt"
server_key: "2c731d63d9ab34028577bad6f5098268"
server_key_iv: "99961ee4be11d423e6e21c5f126d8b47"
server_key_version: "encryptionkey@0"
I20260926 21:35:13.281773 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "03de1f5a91284c87beaee9c3558f62da"
format_stamp: "Formatted at 2026-09-26 21:35:13 on dist-test-slave-87bt"
server_key: "2c731d63d9ab34028577bad6f5098268"
server_key_iv: "99961ee4be11d423e6e21c5f126d8b47"
server_key_version: "encryptionkey@0"
I20260926 21:35:13.286824 20862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:35:13.289943 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:13.291268 20862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:13.291445 20862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "03de1f5a91284c87beaee9c3558f62da"
format_stamp: "Formatted at 2026-09-26 21:35:13 on dist-test-slave-87bt"
server_key: "2c731d63d9ab34028577bad6f5098268"
server_key_iv: "99961ee4be11d423e6e21c5f126d8b47"
server_key_version: "encryptionkey@0"
I20260926 21:35:13.291627 20862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:13.312017 20862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:13.316052 20862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:13.316345 20862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:13.335574 20862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41725
I20260926 21:35:13.335628 20928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41725 every 8 connection(s)
I20260926 21:35:13.337018 20862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:13.339973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 20862
I20260926 21:35:13.340157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:13.340596 388 external_mini_cluster.cc:1428] Setting key 06593749f3811e28af5d90fcdf23a842
I20260926 21:35:13.341161 20929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:13.354019 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:13.342335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45713 (local address 127.0.0.1:41725)
0926 21:35:13.343183 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:13.343195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:13.343235 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:35:13.344773 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:13.344801 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:13.345196 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:13.345537 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:13.346148 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:13.347529 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:13.349487 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:13.350155 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:13.350290 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:13.351635 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:13.351643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:13.351658 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:13.352235 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:13.352319 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:13.352345 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:13.352462 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:13.352763 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 21:35:13.353016 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":370,"threads_started":1}
I20260926 21:35:13.355520 20929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da: Bootstrap starting.
I20260926 21:35:13.357877 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:13.359037 20929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:13.360270 20929 log.cc:824] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:13.363376 20929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da: No bootstrap required, opened a new log
I20260926 21:35:13.370606 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:13.363423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57413 (local address 127.0.0.1:41725)
0926 21:35:13.363675 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:13.363681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:13.363939 (+ 258us) server_negotiation.cc:407] Connection header received
0926 21:35:13.364090 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:13.364095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:13.364178 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:13.364316 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:13.364843 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:13.366052 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:13.367647 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:13.368370 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:13.368500 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:13.369772 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:13.369779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:13.369785 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:13.369806 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:13.369850 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:13.369869 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:13.369983 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:13.370165 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 21:35:13.370223 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260926 21:35:13.373317 20929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03de1f5a91284c87beaee9c3558f62da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260926 21:35:13.373661 20929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:13.373728 20929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03de1f5a91284c87beaee9c3558f62da, State: Initialized, Role: FOLLOWER
I20260926 21:35:13.374290 20929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03de1f5a91284c87beaee9c3558f62da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260926 21:35:13.374573 20929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:13.374739 20929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:13.374948 20929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:13.377602 20929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03de1f5a91284c87beaee9c3558f62da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260926 21:35:13.378139 20929 leader_election.cc:302] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03de1f5a91284c87beaee9c3558f62da; no voters:
I20260926 21:35:13.378770 20929 leader_election.cc:288] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:13.378877 20939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:13.379797 20939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [term 1 LEADER]: Becoming Leader. State: Replica: 03de1f5a91284c87beaee9c3558f62da, State: Running, Role: LEADER
I20260926 21:35:13.380348 20939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03de1f5a91284c87beaee9c3558f62da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260926 21:35:13.380873 20929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:13.383527 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03de1f5a91284c87beaee9c3558f62da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03de1f5a91284c87beaee9c3558f62da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } } }
I20260926 21:35:13.383838 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:13.383001 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03de1f5a91284c87beaee9c3558f62da. Latest consensus state: current_term: 1 leader_uuid: "03de1f5a91284c87beaee9c3558f62da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03de1f5a91284c87beaee9c3558f62da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } } }
I20260926 21:35:13.384501 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:13.387547 20948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:13.392520 20948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:13.401940 20948 catalog_manager.cc:1347] Generated new cluster ID: 70a359a53fcc4428961ab283588ad03a
I20260926 21:35:13.402045 20948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:13.414956 20948 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:13.416728 20948 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:13.427243 20948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03de1f5a91284c87beaee9c3558f62da: Generated new TSK 0
I20260926 21:35:13.428369 20948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:13.444233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20862
W20260926 21:35:13.446127 20924 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45713
W20260926 21:35:13.446959 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:13.447144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 20862
W20260926 21:35:13.463989 20847 connection.cc:535] client connection to 127.0.0.1:41725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41725): Connection reset by peer (error 104)
2026-09-26T21:35:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:13 dist-test-slave-87bt krb5kdc[20961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:13 dist-test-slave-87bt krb5kdc[20961](info): set up 2 sockets
Sep 26 21:35:13 dist-test-slave-87bt krb5kdc[20961](info): commencing operation
krb5kdc: starting...
W20260926 21:35:14.570778 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:14 dist-test-slave-87bt krb5kdc[20961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:14.827378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:15.163431 20974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:15.163890 20974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:15.164067 20974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:15.171433 20974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:15.171597 20974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:15.171701 20974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:15.171820 20974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:15.171907 20974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:15.182139 20974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:15.183486 20974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:15.184886 20974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:15.196699 20982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:15.196645 20980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:15.196758 20979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:15.197690 20974 server_base.cc:1034] running on GCE node
I20260926 21:35:15.198622 20974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:15.200100 20974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:15.201416 20974 hybrid_clock.cc:648] HybridClock initialized: now 1790458515201355 us; error 50 us; skew 500 ppm
I20260926 21:35:15.201810 20974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:15 dist-test-slave-87bt krb5kdc[20961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:15.209398 20974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:15.211138 20974 webserver.cc:460] Webserver started at http://127.0.0.1:32913/ using document root <none> and password file <none>
I20260926 21:35:15.211877 20974 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:15.211983 20974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:15.212240 20974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:15.215344 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "518ed69077bc406c87f88efdea9fe8b3"
format_stamp: "Formatted at 2026-09-26 21:35:15 on dist-test-slave-87bt"
server_key: "260cca02ddc1346d577b79b82bc1e737"
server_key_iv: "8a153ef245e674da807dc0b00d302ec3"
server_key_version: "encryptionkey@0"
I20260926 21:35:15.216065 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "518ed69077bc406c87f88efdea9fe8b3"
format_stamp: "Formatted at 2026-09-26 21:35:15 on dist-test-slave-87bt"
server_key: "260cca02ddc1346d577b79b82bc1e737"
server_key_iv: "8a153ef245e674da807dc0b00d302ec3"
server_key_version: "encryptionkey@0"
I20260926 21:35:15.221621 20974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:35:15.225481 20989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:15.227037 20974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:15.227207 20974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "518ed69077bc406c87f88efdea9fe8b3"
format_stamp: "Formatted at 2026-09-26 21:35:15 on dist-test-slave-87bt"
server_key: "260cca02ddc1346d577b79b82bc1e737"
server_key_iv: "8a153ef245e674da807dc0b00d302ec3"
server_key_version: "encryptionkey@0"
I20260926 21:35:15.227419 20974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:15.233367 20974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:15.237586 20974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:15.237844 20974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:15.256309 20974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43425
I20260926 21:35:15.256345 21040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43425 every 8 connection(s)
I20260926 21:35:15.257742 20974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:15.261797 21041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:15.265751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 20974
I20260926 21:35:15.265965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:15.266480 388 external_mini_cluster.cc:1428] Setting key 0c26e028f7eb1e477d51539201ebcd1d
I20260926 21:35:15.275909 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:15.268515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39735 (local address 127.0.0.1:43425)
0926 21:35:15.269384 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:15.269400 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:15.269455 (+ 55us) server_negotiation.cc:407] Connection header received
0926 21:35:15.271695 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:15.271733 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:15.272240 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:15.272746 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:15.273841 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:15.273850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:15.273868 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:15.273895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:15.273978 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:15.274010 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:15.274257 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:15.274411 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 21:35:15.274642 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":343,"threads_started":1}
I20260926 21:35:15.276644 21041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3: Bootstrap starting.
I20260926 21:35:15.279706 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:15.280493 21041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:15.281608 21041 log.cc:824] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:15.285044 21041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3: No bootstrap required, opened a new log
I20260926 21:35:15.287752 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:15.284387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47493 (local address 127.0.0.1:43425)
0926 21:35:15.284579 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:15.284585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:15.285715 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 21:35:15.285867 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:15.285872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:15.285929 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:15.286038 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:15.287027 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:15.287036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:15.287042 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:15.287068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:15.287112 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:15.287136 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:15.287267 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:15.287367 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 21:35:15.287451 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:35:15.293318 21041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ed69077bc406c87f88efdea9fe8b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260926 21:35:15.293665 21041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:15.293818 21041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518ed69077bc406c87f88efdea9fe8b3, State: Initialized, Role: FOLLOWER
I20260926 21:35:15.294502 21041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ed69077bc406c87f88efdea9fe8b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260926 21:35:15.294749 21041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:15.294903 21041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:15.295083 21041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:15.297264 21041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ed69077bc406c87f88efdea9fe8b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260926 21:35:15.297775 21041 leader_election.cc:302] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 518ed69077bc406c87f88efdea9fe8b3; no voters:
I20260926 21:35:15.298411 21041 leader_election.cc:288] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:15.298585 21051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:15.299381 21051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [term 1 LEADER]: Becoming Leader. State: Replica: 518ed69077bc406c87f88efdea9fe8b3, State: Running, Role: LEADER
I20260926 21:35:15.299947 21051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ed69077bc406c87f88efdea9fe8b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260926 21:35:15.300529 21041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:15.302865 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "518ed69077bc406c87f88efdea9fe8b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ed69077bc406c87f88efdea9fe8b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } } }
I20260926 21:35:15.303237 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:15.303938 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 518ed69077bc406c87f88efdea9fe8b3. Latest consensus state: current_term: 1 leader_uuid: "518ed69077bc406c87f88efdea9fe8b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ed69077bc406c87f88efdea9fe8b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } } }
I20260926 21:35:15.304344 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:15.304838 21059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:15.309276 21059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:15.318493 21059 catalog_manager.cc:1347] Generated new cluster ID: a88639fa6c474dcf9d63d93087a53786
I20260926 21:35:15.318614 21059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:15.333400 21059 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:15.333864 21059 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:15.340975 21059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 518ed69077bc406c87f88efdea9fe8b3: Generated new TSK 0
I20260926 21:35:15.341849 21059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:15.348875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20974
W20260926 21:35:15.349963 21036 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39735
W20260926 21:35:15.350710 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 20974: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:15.350890 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 20974
2026-09-26T21:35:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:15 dist-test-slave-87bt krb5kdc[21072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:15 dist-test-slave-87bt krb5kdc[21072](info): set up 2 sockets
Sep 26 21:35:15 dist-test-slave-87bt krb5kdc[21072](info): commencing operation
krb5kdc: starting...
W20260926 21:35:16.450681 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:16 dist-test-slave-87bt krb5kdc[21072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:16.706117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:17.044016 21085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:17.044384 21085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:17.044559 21085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:17.052073 21085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:17.052242 21085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:17.052363 21085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:17.052477 21085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:17.052558 21085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:17.062950 21085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:17.064459 21085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:17.065837 21085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:17.078148 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:17.078106 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:17.078120 21093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:17.079041 21085 server_base.cc:1034] running on GCE node
I20260926 21:35:17.079896 21085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:17.081310 21085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:17.082584 21085 hybrid_clock.cc:648] HybridClock initialized: now 1790458517082520 us; error 47 us; skew 500 ppm
I20260926 21:35:17.082957 21085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:17 dist-test-slave-87bt krb5kdc[21072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:17.086140 21085 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:17.087648 21085 webserver.cc:460] Webserver started at http://127.0.0.1:41557/ using document root <none> and password file <none>
I20260926 21:35:17.088347 21085 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:17.088495 21085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:17.088814 21085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:17.091830 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "56823a57cfa746e5a36fa88092d60c20"
format_stamp: "Formatted at 2026-09-26 21:35:17 on dist-test-slave-87bt"
server_key: "8d583e7d470713f9b1b5f2cda5b3a7ba"
server_key_iv: "23a7e8f2176fec1839a6237aac34d749"
server_key_version: "encryptionkey@0"
I20260926 21:35:17.092692 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "56823a57cfa746e5a36fa88092d60c20"
format_stamp: "Formatted at 2026-09-26 21:35:17 on dist-test-slave-87bt"
server_key: "8d583e7d470713f9b1b5f2cda5b3a7ba"
server_key_iv: "23a7e8f2176fec1839a6237aac34d749"
server_key_version: "encryptionkey@0"
I20260926 21:35:17.097723 21085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 21:35:17.101141 21100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:17.102502 21085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:35:17.102687 21085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "56823a57cfa746e5a36fa88092d60c20"
format_stamp: "Formatted at 2026-09-26 21:35:17 on dist-test-slave-87bt"
server_key: "8d583e7d470713f9b1b5f2cda5b3a7ba"
server_key_iv: "23a7e8f2176fec1839a6237aac34d749"
server_key_version: "encryptionkey@0"
I20260926 21:35:17.102890 21085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:17.121415 21085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:17.125510 21085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:17.125830 21085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:17.144933 21085 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42649
I20260926 21:35:17.144958 21151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42649 every 8 connection(s)
I20260926 21:35:17.146981 21085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:17.150943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21085
I20260926 21:35:17.151122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:17.151669 388 external_mini_cluster.cc:1428] Setting key a77214576d2d39d39b9fd8e78f998d90
I20260926 21:35:17.152235 21152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:17.167394 21152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20: Bootstrap starting.
Sep 26 21:35:17 dist-test-slave-87bt krb5kdc[21072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:35:17.170516 21152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:17.171571 21152 log.cc:824] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:17.174285 21152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20: No bootstrap required, opened a new log
I20260926 21:35:17.175796 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:17.153343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35059 (local address 127.0.0.1:42649)
0926 21:35:17.154296 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:17.154318 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:17.155031 (+ 713us) server_negotiation.cc:407] Connection header received
0926 21:35:17.157491 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:17.157513 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:17.157914 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:17.158258 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:17.160041 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:17.161316 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:17.163200 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:17.163889 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:17.164026 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:17.168814 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:17.168821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:17.168838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:17.168876 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:17.171960 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:17.172628 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:17.172635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:17.172644 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:17.172801 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:17.173188 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:17.173193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:17.173195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:17.173684 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:17.173989 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:17.174475 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 21:35:17.174876 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":370,"threads_started":1}
I20260926 21:35:17.179350 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:17.182477 21152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56823a57cfa746e5a36fa88092d60c20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42649 } }
I20260926 21:35:17.182896 21152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:17.183001 21152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56823a57cfa746e5a36fa88092d60c20, State: Initialized, Role: FOLLOWER
I20260926 21:35:17.183704 21152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56823a57cfa746e5a36fa88092d60c20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42649 } }
I20260926 21:35:17.184039 21152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:17.184207 21152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:17.184401 21152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:17.186801 21152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56823a57cfa746e5a36fa88092d60c20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42649 } }
I20260926 21:35:17.187314 21152 leader_election.cc:302] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56823a57cfa746e5a36fa88092d60c20; no voters:
I20260926 21:35:17.187976 21152 leader_election.cc:288] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:17.188081 21162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:17.190625 21152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:17.195187 21162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [term 1 LEADER]: Becoming Leader. State: Replica: 56823a57cfa746e5a36fa88092d60c20, State: Running, Role: LEADER
I20260926 21:35:17.196010 21162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56823a57cfa746e5a36fa88092d60c20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42649 } }
I20260926 21:35:17.197753 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:17.183562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55597 (local address 127.0.0.1:42649)
0926 21:35:17.183749 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:17.183756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:17.184351 (+ 595us) server_negotiation.cc:407] Connection header received
0926 21:35:17.184538 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:17.184543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:17.184617 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:17.184733 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:17.186061 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:17.186851 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:17.188309 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:17.189020 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:17.189135 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:17.191743 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:17.191755 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:17.191759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:17.191789 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:17.194968 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:17.195725 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:17.195730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:17.195734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:17.195883 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:17.196280 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:17.196285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:17.196288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:17.196638 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:17.196903 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:17.197222 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 21:35:17.197431 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:35:17.199779 21163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56823a57cfa746e5a36fa88092d60c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56823a57cfa746e5a36fa88092d60c20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42649 } } }
I20260926 21:35:17.200275 21163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:17.202723 21164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56823a57cfa746e5a36fa88092d60c20. Latest consensus state: current_term: 1 leader_uuid: "56823a57cfa746e5a36fa88092d60c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56823a57cfa746e5a36fa88092d60c20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42649 } } }
I20260926 21:35:17.203058 21164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:17.203769 21170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:17.209023 21170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:17.217796 21170 catalog_manager.cc:1347] Generated new cluster ID: 421e9286e07845ffa115e634ddc6cb73
I20260926 21:35:17.217933 21170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:17.233918 21170 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:17.235891 21170 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:17.245255 21170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56823a57cfa746e5a36fa88092d60c20: Generated new TSK 0
I20260926 21:35:17.246347 21170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:17.257619 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21085
I20260926 21:35:17.258672 21147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:17.352560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21085
2026-09-26T21:35:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:17 dist-test-slave-87bt krb5kdc[21185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:17 dist-test-slave-87bt krb5kdc[21185](info): set up 2 sockets
Sep 26 21:35:17 dist-test-slave-87bt krb5kdc[21185](info): commencing operation
krb5kdc: starting...
W20260926 21:35:18.446895 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:18 dist-test-slave-87bt krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:18.701086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:19.035156 21198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:19.035454 21198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:19.035626 21198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:19.043007 21198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:19.043133 21198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:19.043241 21198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:19.043355 21198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:19.043437 21198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:19.053871 21198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:19.055264 21198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:19.056638 21198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:19.069262 21203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:19.069294 21206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:19.069257 21204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:19.070173 21198 server_base.cc:1034] running on GCE node
I20260926 21:35:19.071067 21198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:19.072527 21198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:19.073734 21198 hybrid_clock.cc:648] HybridClock initialized: now 1790458519073695 us; error 36 us; skew 500 ppm
I20260926 21:35:19.074095 21198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:19 dist-test-slave-87bt krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:19.077217 21198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:19.078724 21198 webserver.cc:460] Webserver started at http://127.0.0.1:35963/ using document root <none> and password file <none>
I20260926 21:35:19.079433 21198 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:19.079576 21198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:19.079890 21198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:19.082950 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "31a6fc5d205d4a32aa610d1b9602c49e"
format_stamp: "Formatted at 2026-09-26 21:35:19 on dist-test-slave-87bt"
server_key: "419474e160bd0318a38366ecc099a126"
server_key_iv: "40d8e804af619c0ebc3ec6b2deac01f3"
server_key_version: "encryptionkey@0"
I20260926 21:35:19.083715 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "31a6fc5d205d4a32aa610d1b9602c49e"
format_stamp: "Formatted at 2026-09-26 21:35:19 on dist-test-slave-87bt"
server_key: "419474e160bd0318a38366ecc099a126"
server_key_iv: "40d8e804af619c0ebc3ec6b2deac01f3"
server_key_version: "encryptionkey@0"
I20260926 21:35:19.089051 21198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260926 21:35:19.092629 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:19.093990 21198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 21:35:19.094169 21198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "31a6fc5d205d4a32aa610d1b9602c49e"
format_stamp: "Formatted at 2026-09-26 21:35:19 on dist-test-slave-87bt"
server_key: "419474e160bd0318a38366ecc099a126"
server_key_iv: "40d8e804af619c0ebc3ec6b2deac01f3"
server_key_version: "encryptionkey@0"
I20260926 21:35:19.094384 21198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:19.121119 21198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:19.125320 21198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:19.125622 21198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:19.144711 21198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36241
I20260926 21:35:19.144730 21264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36241 every 8 connection(s)
I20260926 21:35:19.146132 21198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:19.150167 21265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:19.156208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21198
I20260926 21:35:19.156400 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:19.156857 388 external_mini_cluster.cc:1428] Setting key 6bbe5ecb4a97293289a94cc6eab38b0c
I20260926 21:35:19.162276 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e: Bootstrap starting.
I20260926 21:35:19.166231 21265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:19.167397 21265 log.cc:824] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:19.170354 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e: No bootstrap required, opened a new log
I20260926 21:35:19.172240 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:19.158783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54083 (local address 127.0.0.1:36241)
0926 21:35:19.159668 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:19.159685 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:19.159737 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:35:19.162124 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:19.162152 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:19.162701 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:19.163155 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:19.163666 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:19.165714 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:19.167208 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:19.168248 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:19.168472 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:19.169621 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:19.169628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:19.169643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:19.170166 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:19.170241 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:19.170265 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:19.170408 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:19.170650 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 21:35:19.170965 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":393,"threads_started":1}
I20260926 21:35:19.176319 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:19.179136 21265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a6fc5d205d4a32aa610d1b9602c49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } }
I20260926 21:35:19.179507 21265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:19.179629 21265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a6fc5d205d4a32aa610d1b9602c49e, State: Initialized, Role: FOLLOWER
I20260926 21:35:19.180182 21265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a6fc5d205d4a32aa610d1b9602c49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } }
I20260926 21:35:19.180439 21265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:19.180604 21265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:19.180833 21265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:19.183964 21265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a6fc5d205d4a32aa610d1b9602c49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } }
I20260926 21:35:19.184547 21265 leader_election.cc:302] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31a6fc5d205d4a32aa610d1b9602c49e; no voters:
I20260926 21:35:19.185278 21265 leader_election.cc:288] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:19.185403 21275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:19.186525 21275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [term 1 LEADER]: Becoming Leader. State: Replica: 31a6fc5d205d4a32aa610d1b9602c49e, State: Running, Role: LEADER
I20260926 21:35:19.187201 21275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a6fc5d205d4a32aa610d1b9602c49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } }
I20260926 21:35:19.187575 21265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:19.188740 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:19.180787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49665 (local address 127.0.0.1:36241)
0926 21:35:19.180998 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:19.181006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:19.181433 (+ 427us) server_negotiation.cc:407] Connection header received
0926 21:35:19.181634 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:19.181641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:19.181704 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:19.181785 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:19.182375 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:19.183136 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:19.184975 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:19.185853 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:19.186771 (+ 918us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:19.187429 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:19.187437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:19.187442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:19.187464 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:19.187510 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:19.187529 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:19.188102 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:19.188147 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 21:35:19.188256 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 21:35:19.196244 21276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31a6fc5d205d4a32aa610d1b9602c49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a6fc5d205d4a32aa610d1b9602c49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } } }
I20260926 21:35:19.196707 21276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:19.197208 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31a6fc5d205d4a32aa610d1b9602c49e. Latest consensus state: current_term: 1 leader_uuid: "31a6fc5d205d4a32aa610d1b9602c49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a6fc5d205d4a32aa610d1b9602c49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } } }
I20260926 21:35:19.197458 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:19.203078 21285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:19.208590 21285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:19.217561 21285 catalog_manager.cc:1347] Generated new cluster ID: 07f524ea672f4280988275d5f2768fae
I20260926 21:35:19.217702 21285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:19.252499 21285 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:19.253904 21285 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:19.261340 21285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31a6fc5d205d4a32aa610d1b9602c49e: Generated new TSK 0
I20260926 21:35:19.262189 21285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:19.278002 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21198
W20260926 21:35:19.279102 21260 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54083
W20260926 21:35:19.279821 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21198: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:19.279983 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21198
2026-09-26T21:35:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:19 dist-test-slave-87bt krb5kdc[21296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:19 dist-test-slave-87bt krb5kdc[21296](info): set up 2 sockets
Sep 26 21:35:19 dist-test-slave-87bt krb5kdc[21296](info): commencing operation
krb5kdc: starting...
W20260926 21:35:20.390996 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:20 dist-test-slave-87bt krb5kdc[21296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:20.649523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:21.001201 21309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:21.001566 21309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:21.001758 21309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:21.009254 21309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:21.009378 21309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:21.009425 21309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:21.009478 21309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:21.009531 21309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:21.019861 21309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:21.021183 21309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:21.022567 21309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:21.034503 21317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:21.034612 21315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:21.034574 21314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:21.036414 21309 server_base.cc:1034] running on GCE node
I20260926 21:35:21.037272 21309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:21.038734 21309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:21.039999 21309 hybrid_clock.cc:648] HybridClock initialized: now 1790458521039942 us; error 36 us; skew 500 ppm
I20260926 21:35:21.040444 21309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:21 dist-test-slave-87bt krb5kdc[21296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:21.045104 21309 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:21.047418 21309 webserver.cc:460] Webserver started at http://127.0.0.1:42357/ using document root <none> and password file <none>
I20260926 21:35:21.048342 21309 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:21.048485 21309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:21.048800 21309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:21.052872 21309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "339af98b60814ba7a0e0c6623ebf3e02"
format_stamp: "Formatted at 2026-09-26 21:35:21 on dist-test-slave-87bt"
server_key: "7a5ce8463a0c24f828e6e3ead7342673"
server_key_iv: "348866b8105666058459c94630c618fd"
server_key_version: "encryptionkey@0"
I20260926 21:35:21.053782 21309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "339af98b60814ba7a0e0c6623ebf3e02"
format_stamp: "Formatted at 2026-09-26 21:35:21 on dist-test-slave-87bt"
server_key: "7a5ce8463a0c24f828e6e3ead7342673"
server_key_iv: "348866b8105666058459c94630c618fd"
server_key_version: "encryptionkey@0"
I20260926 21:35:21.059414 21309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 21:35:21.062996 21324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:21.064595 21309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:35:21.064759 21309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "339af98b60814ba7a0e0c6623ebf3e02"
format_stamp: "Formatted at 2026-09-26 21:35:21 on dist-test-slave-87bt"
server_key: "7a5ce8463a0c24f828e6e3ead7342673"
server_key_iv: "348866b8105666058459c94630c618fd"
server_key_version: "encryptionkey@0"
I20260926 21:35:21.064939 21309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:21.070935 21309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:21.075206 21309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:21.075453 21309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:21.094215 21309 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32981
I20260926 21:35:21.094319 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32981 every 8 connection(s)
I20260926 21:35:21.095633 21309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:21.099310 21376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:21.104315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21309
I20260926 21:35:21.104509 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:21.105073 388 external_mini_cluster.cc:1428] Setting key 5076c26c10260ed202ccc9c0fd1e0c59
I20260926 21:35:21.111446 21376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02: Bootstrap starting.
I20260926 21:35:21.113169 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:21.106741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54433 (local address 127.0.0.1:32981)
0926 21:35:21.107522 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:21.107533 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:21.107574 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:35:21.109817 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:21.109841 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:21.110198 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:21.110603 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:21.111642 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:21.111648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:21.111663 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:21.111676 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:21.111733 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:21.111760 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:21.111920 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:21.112060 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 21:35:21.112285 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":370,"threads_started":1}
I20260926 21:35:21.114890 21376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:21.116137 21376 log.cc:824] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:21.117043 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:21.119095 21376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02: No bootstrap required, opened a new log
I20260926 21:35:21.123932 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:21.121468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45085 (local address 127.0.0.1:32981)
0926 21:35:21.121666 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:21.121674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:21.121987 (+ 313us) server_negotiation.cc:407] Connection header received
0926 21:35:21.122196 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:21.122200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:21.122250 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:21.122359 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:21.123267 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:21.123273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:21.123277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:21.123295 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:21.123344 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:21.123359 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:21.123492 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:21.123582 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 21:35:21.123646 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:35:21.127727 21376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339af98b60814ba7a0e0c6623ebf3e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260926 21:35:21.128125 21376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:21.128247 21376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339af98b60814ba7a0e0c6623ebf3e02, State: Initialized, Role: FOLLOWER
I20260926 21:35:21.128856 21376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339af98b60814ba7a0e0c6623ebf3e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260926 21:35:21.129108 21376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:21.129263 21376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:21.129453 21376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:21.132102 21376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339af98b60814ba7a0e0c6623ebf3e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260926 21:35:21.132618 21376 leader_election.cc:302] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 339af98b60814ba7a0e0c6623ebf3e02; no voters:
I20260926 21:35:21.133215 21376 leader_election.cc:288] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:21.133355 21386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:21.134040 21386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [term 1 LEADER]: Becoming Leader. State: Replica: 339af98b60814ba7a0e0c6623ebf3e02, State: Running, Role: LEADER
I20260926 21:35:21.134606 21386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339af98b60814ba7a0e0c6623ebf3e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260926 21:35:21.135412 21376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:21.137177 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 339af98b60814ba7a0e0c6623ebf3e02. Latest consensus state: current_term: 1 leader_uuid: "339af98b60814ba7a0e0c6623ebf3e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339af98b60814ba7a0e0c6623ebf3e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } } }
I20260926 21:35:21.137702 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:21.137702 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "339af98b60814ba7a0e0c6623ebf3e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339af98b60814ba7a0e0c6623ebf3e02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } } }
I20260926 21:35:21.138028 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:21.140513 21391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:21.145740 21391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:21.155380 21391 catalog_manager.cc:1347] Generated new cluster ID: 34499861b1864ae48ff7166f09fc528c
I20260926 21:35:21.155527 21391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:21.195894 21391 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:21.196343 21391 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:21.206244 21391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 339af98b60814ba7a0e0c6623ebf3e02: Generated new TSK 0
I20260926 21:35:21.207134 21391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:21.225590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21309
W20260926 21:35:21.226754 21371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54433
W20260926 21:35:21.227515 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21309: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:21.227680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21309
2026-09-26T21:35:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:21 dist-test-slave-87bt krb5kdc[21407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:21 dist-test-slave-87bt krb5kdc[21407](info): set up 2 sockets
Sep 26 21:35:21 dist-test-slave-87bt krb5kdc[21407](info): commencing operation
krb5kdc: starting...
W20260926 21:35:22.338915 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:22 dist-test-slave-87bt krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:22.597601 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:22.930212 21420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:22.930641 21420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:22.930845 21420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:22.938351 21420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:22.938571 21420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:22.938679 21420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:22.938769 21420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:22.938833 21420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:22.949153 21420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:22.950551 21420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:22.951936 21420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:22.963863 21425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:22.964043 21426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:22.964475 21428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:22.964990 21420 server_base.cc:1034] running on GCE node
I20260926 21:35:22.965782 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:22.967284 21420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:22.968520 21420 hybrid_clock.cc:648] HybridClock initialized: now 1790458522968474 us; error 36 us; skew 500 ppm
I20260926 21:35:22.968883 21420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:22 dist-test-slave-87bt krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:22.972028 21420 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:22.973609 21420 webserver.cc:460] Webserver started at http://127.0.0.1:39179/ using document root <none> and password file <none>
I20260926 21:35:22.974334 21420 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:22.974530 21420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:22.974858 21420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:22.977878 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "439c1ea5c7904544909c8e0dc3249ed7"
format_stamp: "Formatted at 2026-09-26 21:35:22 on dist-test-slave-87bt"
server_key: "0f5f9bfd9830c96631dcb6efe5124a24"
server_key_iv: "108eb567d5775009745f212833e868c3"
server_key_version: "encryptionkey@0"
I20260926 21:35:22.978730 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "439c1ea5c7904544909c8e0dc3249ed7"
format_stamp: "Formatted at 2026-09-26 21:35:22 on dist-test-slave-87bt"
server_key: "0f5f9bfd9830c96631dcb6efe5124a24"
server_key_iv: "108eb567d5775009745f212833e868c3"
server_key_version: "encryptionkey@0"
I20260926 21:35:22.983971 21420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:35:22.987534 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:22.988848 21420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:35:22.989068 21420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "439c1ea5c7904544909c8e0dc3249ed7"
format_stamp: "Formatted at 2026-09-26 21:35:22 on dist-test-slave-87bt"
server_key: "0f5f9bfd9830c96631dcb6efe5124a24"
server_key_iv: "108eb567d5775009745f212833e868c3"
server_key_version: "encryptionkey@0"
I20260926 21:35:22.989275 21420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:23.008836 21420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:23.013615 21420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:23.013935 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:23.033174 21420 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36505
I20260926 21:35:23.033203 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36505 every 8 connection(s)
I20260926 21:35:23.034682 21420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:23.038592 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:23.041831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21420
I20260926 21:35:23.042026 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:23.042588 388 external_mini_cluster.cc:1428] Setting key 2575b1d7b21ae34c1bf69cc5cf38600e
I20260926 21:35:23.051024 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7: Bootstrap starting.
I20260926 21:35:23.054402 21487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:23.055518 21487 log.cc:824] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:35:23 dist-test-slave-87bt krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:35:23.059145 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7: No bootstrap required, opened a new log
I20260926 21:35:23.068235 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:23.044759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:47449 (local address 127.0.0.1:36505)
0926 21:35:23.045540 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:23.045553 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:23.045596 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:35:23.047424 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:23.047450 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:23.047916 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:23.048357 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:23.049827 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:23.051341 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:23.052656 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:23.053361 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:23.053570 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:23.058365 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:23.058382 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:23.058404 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:23.058496 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:23.062739 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:23.063747 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:23.063758 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:23.063768 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:23.064009 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:23.064551 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:23.064557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:23.064562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:23.065405 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:23.065767 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:23.066236 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 21:35:23.066685 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":331,"threads_started":1}
I20260926 21:35:23.070621 21487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c1ea5c7904544909c8e0dc3249ed7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } }
I20260926 21:35:23.071111 21487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:23.071228 21487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439c1ea5c7904544909c8e0dc3249ed7, State: Initialized, Role: FOLLOWER
I20260926 21:35:23.071873 21487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c1ea5c7904544909c8e0dc3249ed7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } }
I20260926 21:35:23.072211 21487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:23.072372 21487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:23.072563 21487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:23.072954 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:23.074996 21487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c1ea5c7904544909c8e0dc3249ed7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } }
I20260926 21:35:23.075603 21487 leader_election.cc:302] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439c1ea5c7904544909c8e0dc3249ed7; no voters:
I20260926 21:35:23.076306 21487 leader_election.cc:288] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:23.076422 21495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:23.078130 21495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [term 1 LEADER]: Becoming Leader. State: Replica: 439c1ea5c7904544909c8e0dc3249ed7, State: Running, Role: LEADER
I20260926 21:35:23.078699 21487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:23.078743 21495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c1ea5c7904544909c8e0dc3249ed7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } }
I20260926 21:35:23.088328 21498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439c1ea5c7904544909c8e0dc3249ed7. Latest consensus state: current_term: 1 leader_uuid: "439c1ea5c7904544909c8e0dc3249ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c1ea5c7904544909c8e0dc3249ed7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } } }
I20260926 21:35:23.088394 21497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "439c1ea5c7904544909c8e0dc3249ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c1ea5c7904544909c8e0dc3249ed7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } } }
I20260926 21:35:23.088869 21497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:23.088871 21498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:23.089761 21505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:23.096035 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:23.077536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:54445 (local address 127.0.0.1:36505)
0926 21:35:23.077697 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:23.077706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:23.078218 (+ 512us) server_negotiation.cc:407] Connection header received
0926 21:35:23.078402 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:23.078407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:23.078524 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:23.078606 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:23.080321 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:23.081221 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:23.082535 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:23.083036 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:23.083190 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:23.085734 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:23.085739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:23.085742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:23.085776 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:23.090227 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:23.091690 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:23.091698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:23.091704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:23.091906 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:23.092706 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:23.092714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:23.092721 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:23.093274 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:23.095000 (+ 1726us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:23.095239 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 21:35:23.095480 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 21:35:23.104447 21505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:23.115187 21505 catalog_manager.cc:1347] Generated new cluster ID: e82baeeabfe847bdb834144907ceedd3
I20260926 21:35:23.115312 21505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:23.158764 21505 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:23.160116 21505 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:23.169449 21505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439c1ea5c7904544909c8e0dc3249ed7: Generated new TSK 0
I20260926 21:35:23.170379 21505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:23.181031 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21420
I20260926 21:35:23.182212 21482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:23.281574 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21420
W20260926 21:35:23.298861 21405 connection.cc:535] client connection to 127.0.0.1:36505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36505): Connection reset by peer (error 104)
2026-09-26T21:35:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:23 dist-test-slave-87bt krb5kdc[21519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:23 dist-test-slave-87bt krb5kdc[21519](info): set up 2 sockets
Sep 26 21:35:23 dist-test-slave-87bt krb5kdc[21519](info): commencing operation
krb5kdc: starting...
W20260926 21:35:24.383068 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:24 dist-test-slave-87bt krb5kdc[21519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:24.645920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:24.979444 21532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:24.979737 21532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:24.979828 21532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:24.986930 21532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:24.987038 21532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:24.987092 21532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:24.987154 21532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:24.987197 21532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:24.997031 21532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:24.998391 21532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:24.999670 21532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:25.011404 21538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:25.011492 21540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:25.011492 21537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:25.012288 21532 server_base.cc:1034] running on GCE node
I20260926 21:35:25.013100 21532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:25.014544 21532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:25.015774 21532 hybrid_clock.cc:648] HybridClock initialized: now 1790458525015733 us; error 50 us; skew 500 ppm
I20260926 21:35:25.016125 21532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:25 dist-test-slave-87bt krb5kdc[21519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:25.019253 21532 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:25.020632 21532 webserver.cc:460] Webserver started at http://127.0.0.1:38583/ using document root <none> and password file <none>
I20260926 21:35:25.021296 21532 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:25.021394 21532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:25.021656 21532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:25.024703 21532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "fc7c22d33bd342bd843dd1ccc49cebb5"
format_stamp: "Formatted at 2026-09-26 21:35:25 on dist-test-slave-87bt"
server_key: "0eb923597d40dc7754498a0c343dc0c0"
server_key_iv: "1dec44d65afe1593dc08aed147be0f5c"
server_key_version: "encryptionkey@0"
I20260926 21:35:25.025457 21532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc7c22d33bd342bd843dd1ccc49cebb5"
format_stamp: "Formatted at 2026-09-26 21:35:25 on dist-test-slave-87bt"
server_key: "0eb923597d40dc7754498a0c343dc0c0"
server_key_iv: "1dec44d65afe1593dc08aed147be0f5c"
server_key_version: "encryptionkey@0"
I20260926 21:35:25.030642 21532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:35:25.034374 21547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:25.035877 21532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:25.036070 21532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "fc7c22d33bd342bd843dd1ccc49cebb5"
format_stamp: "Formatted at 2026-09-26 21:35:25 on dist-test-slave-87bt"
server_key: "0eb923597d40dc7754498a0c343dc0c0"
server_key_iv: "1dec44d65afe1593dc08aed147be0f5c"
server_key_version: "encryptionkey@0"
I20260926 21:35:25.036278 21532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:25.050354 21532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:25.054567 21532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:25.054862 21532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:25.073326 21532 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38413
I20260926 21:35:25.073356 21598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38413 every 8 connection(s)
I20260926 21:35:25.074827 21532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:25.078718 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:25.079826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21532
I20260926 21:35:25.080000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:25.080417 388 external_mini_cluster.cc:1428] Setting key 24930973576af65d7e63a0261e17eaea
I20260926 21:35:25.093662 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5: Bootstrap starting.
I20260926 21:35:25.095255 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:25.082154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:46797 (local address 127.0.0.1:38413)
0926 21:35:25.082920 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:25.082935 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:25.082983 (+ 48us) server_negotiation.cc:407] Connection header received
0926 21:35:25.085312 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:25.085343 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:25.085941 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:25.086461 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:25.086967 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:25.088825 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:25.090165 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:25.091318 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:25.091499 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:25.092703 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:25.092713 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:25.092734 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:25.093460 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:25.093563 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:25.093593 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:25.093757 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:25.094045 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 21:35:25.094211 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":304,"threads_started":1}
I20260926 21:35:25.097218 21599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:25.098496 21599 log.cc:824] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:25.099321 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:25.101287 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5: No bootstrap required, opened a new log
I20260926 21:35:25.110464 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:25.103742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:40267 (local address 127.0.0.1:38413)
0926 21:35:25.103885 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:25.103889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:25.104296 (+ 407us) server_negotiation.cc:407] Connection header received
0926 21:35:25.104464 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:25.104468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:25.104528 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:25.104607 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:25.105249 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:25.105995 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:25.107926 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:25.108371 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:25.108460 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:25.109670 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:25.109675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:25.109678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:25.109695 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:25.109733 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:25.109747 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:25.109829 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:25.110032 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 21:35:25.110074 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 21:35:25.111367 21599 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7c22d33bd342bd843dd1ccc49cebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260926 21:35:25.111853 21599 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:25.111974 21599 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7c22d33bd342bd843dd1ccc49cebb5, State: Initialized, Role: FOLLOWER
I20260926 21:35:25.112535 21599 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7c22d33bd342bd843dd1ccc49cebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260926 21:35:25.112782 21599 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:25.112946 21599 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:25.113133 21599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:25.115384 21599 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7c22d33bd342bd843dd1ccc49cebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260926 21:35:25.115967 21599 leader_election.cc:302] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc7c22d33bd342bd843dd1ccc49cebb5; no voters:
I20260926 21:35:25.116637 21599 leader_election.cc:288] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:25.116761 21609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:25.117578 21609 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [term 1 LEADER]: Becoming Leader. State: Replica: fc7c22d33bd342bd843dd1ccc49cebb5, State: Running, Role: LEADER
I20260926 21:35:25.118142 21609 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7c22d33bd342bd843dd1ccc49cebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260926 21:35:25.118865 21599 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:25.121227 21610 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc7c22d33bd342bd843dd1ccc49cebb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7c22d33bd342bd843dd1ccc49cebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } } }
I20260926 21:35:25.121711 21610 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:25.122637 21618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:25.123894 21611 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc7c22d33bd342bd843dd1ccc49cebb5. Latest consensus state: current_term: 1 leader_uuid: "fc7c22d33bd342bd843dd1ccc49cebb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7c22d33bd342bd843dd1ccc49cebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } } }
I20260926 21:35:25.124366 21611 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:25.134385 21618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:25.143748 21618 catalog_manager.cc:1347] Generated new cluster ID: 18617cca7fea469399e19a98349363c5
I20260926 21:35:25.143872 21618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:25.156994 21618 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:25.158988 21618 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:25.165988 21618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc7c22d33bd342bd843dd1ccc49cebb5: Generated new TSK 0
I20260926 21:35:25.167066 21618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:25.176635 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21532
W20260926 21:35:25.177793 21594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.94:46797
W20260926 21:35:25.178586 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21532: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:25.178740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21532
W20260926 21:35:25.195837 21517 connection.cc:535] client connection to 127.0.0.1:38413 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38413): Connection reset by peer (error 104)
2026-09-26T21:35:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:25 dist-test-slave-87bt krb5kdc[21630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:25 dist-test-slave-87bt krb5kdc[21630](info): set up 2 sockets
Sep 26 21:35:25 dist-test-slave-87bt krb5kdc[21630](info): commencing operation
krb5kdc: starting...
W20260926 21:35:26.278872 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:26 dist-test-slave-87bt krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:26.545190 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:26.881369 21643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:26.881661 21643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:26.881850 21643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:26.889061 21643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:26.889173 21643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:26.889286 21643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:26.889400 21643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:26.889479 21643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:26.899319 21643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:26.900607 21643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:26.901911 21643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:26.914367 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:26.914502 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:26.914367 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:26.916687 21643 server_base.cc:1034] running on GCE node
I20260926 21:35:26.917629 21643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:26.919220 21643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:26.920426 21643 hybrid_clock.cc:648] HybridClock initialized: now 1790458526920371 us; error 44 us; skew 500 ppm
I20260926 21:35:26.920800 21643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:26 dist-test-slave-87bt krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:26.928200 21643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:26.929986 21643 webserver.cc:460] Webserver started at http://127.0.0.1:42897/ using document root <none> and password file <none>
I20260926 21:35:26.930889 21643 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:26.931048 21643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:26.931376 21643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:26.935534 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "b3a03263abdf4424bc921622e114bcda"
format_stamp: "Formatted at 2026-09-26 21:35:26 on dist-test-slave-87bt"
server_key: "b3ae5bba48468998cdf26af9ebe61744"
server_key_iv: "2b65c23ce57ca2704b0965655875504c"
server_key_version: "encryptionkey@0"
I20260926 21:35:26.936400 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "b3a03263abdf4424bc921622e114bcda"
format_stamp: "Formatted at 2026-09-26 21:35:26 on dist-test-slave-87bt"
server_key: "b3ae5bba48468998cdf26af9ebe61744"
server_key_iv: "2b65c23ce57ca2704b0965655875504c"
server_key_version: "encryptionkey@0"
I20260926 21:35:26.941847 21643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:35:26.945501 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:26.946957 21643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:26.947157 21643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "b3a03263abdf4424bc921622e114bcda"
format_stamp: "Formatted at 2026-09-26 21:35:26 on dist-test-slave-87bt"
server_key: "b3ae5bba48468998cdf26af9ebe61744"
server_key_iv: "2b65c23ce57ca2704b0965655875504c"
server_key_version: "encryptionkey@0"
I20260926 21:35:26.947381 21643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:26.953402 21643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:26.957490 21643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:26.957858 21643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:26.976130 21643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38437
I20260926 21:35:26.976212 21709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38437 every 8 connection(s)
I20260926 21:35:26.977545 21643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:26.980103 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21643
I20260926 21:35:26.980293 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:26.980687 388 external_mini_cluster.cc:1428] Setting key 99847190626ca3b2e7d840d3c1cc3d6e
I20260926 21:35:26.981492 21710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:26.988287 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:26.982416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:54061 (local address 127.0.0.1:38437)
0926 21:35:26.983181 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:26.983193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:26.983232 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:35:26.984792 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:26.984813 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:26.985179 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:26.985583 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:26.986658 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:26.986664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:26.986677 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:26.986691 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:26.986763 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:26.986781 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:26.987031 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:26.987177 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 21:35:26.987366 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":311,"threads_started":1}
I20260926 21:35:26.991856 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:26.993901 21710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda: Bootstrap starting.
I20260926 21:35:26.997479 21710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:26.998814 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:26.996185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:46513 (local address 127.0.0.1:38437)
0926 21:35:26.996392 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:26.996399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:26.996717 (+ 318us) server_negotiation.cc:407] Connection header received
0926 21:35:26.996882 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:26.996886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:26.996931 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:26.997010 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:26.998086 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:26.998090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:26.998094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:26.998108 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:26.998148 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:26.998163 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:26.998255 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:26.998401 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 21:35:26.998505 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 21:35:26.998951 21710 log.cc:824] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:27.001833 21710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda: No bootstrap required, opened a new log
I20260926 21:35:27.010025 21710 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a03263abdf4424bc921622e114bcda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260926 21:35:27.010463 21710 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:27.010620 21710 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a03263abdf4424bc921622e114bcda, State: Initialized, Role: FOLLOWER
I20260926 21:35:27.011197 21710 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a03263abdf4424bc921622e114bcda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260926 21:35:27.011390 21710 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:27.011487 21710 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:27.011620 21710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:27.013688 21710 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a03263abdf4424bc921622e114bcda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260926 21:35:27.014132 21710 leader_election.cc:302] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3a03263abdf4424bc921622e114bcda; no voters:
I20260926 21:35:27.014775 21710 leader_election.cc:288] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:27.014909 21720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:27.015645 21720 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [term 1 LEADER]: Becoming Leader. State: Replica: b3a03263abdf4424bc921622e114bcda, State: Running, Role: LEADER
I20260926 21:35:27.016206 21720 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a03263abdf4424bc921622e114bcda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260926 21:35:27.016981 21710 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:27.019183 21722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3a03263abdf4424bc921622e114bcda. Latest consensus state: current_term: 1 leader_uuid: "b3a03263abdf4424bc921622e114bcda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a03263abdf4424bc921622e114bcda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } } }
I20260926 21:35:27.019696 21722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:27.020588 21727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:27.021694 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3a03263abdf4424bc921622e114bcda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a03263abdf4424bc921622e114bcda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } } }
I20260926 21:35:27.022037 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:27.026787 21727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:27.036084 21727 catalog_manager.cc:1347] Generated new cluster ID: 2309bec7e322416cafc92faea3ecdb91
I20260926 21:35:27.036212 21727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:27.053282 21727 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:27.053784 21727 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:27.068193 21727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3a03263abdf4424bc921622e114bcda: Generated new TSK 0
I20260926 21:35:27.069136 21727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:27.080106 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21643
W20260926 21:35:27.081336 21705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.94:54061
W20260926 21:35:27.082090 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21643: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:27.082237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21643
2026-09-26T21:35:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:27 dist-test-slave-87bt krb5kdc[21741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:27 dist-test-slave-87bt krb5kdc[21741](info): set up 2 sockets
Sep 26 21:35:27 dist-test-slave-87bt krb5kdc[21741](info): commencing operation
krb5kdc: starting...
W20260926 21:35:28.194840 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:28 dist-test-slave-87bt krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:28.453116 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:28.780252 21754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:28.780541 21754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:28.780637 21754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:28.787881 21754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:28.787987 21754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:28.788034 21754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:28.788095 21754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:28.788138 21754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:28.797942 21754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:28.799192 21754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:28.800405 21754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:28.811833 21759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:28.811969 21762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:28.811939 21760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:28.812878 21754 server_base.cc:1034] running on GCE node
I20260926 21:35:28.813638 21754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:28.815163 21754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:28.816391 21754 hybrid_clock.cc:648] HybridClock initialized: now 1790458528816342 us; error 59 us; skew 500 ppm
I20260926 21:35:28.816745 21754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:28 dist-test-slave-87bt krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:28.819824 21754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:28.821324 21754 webserver.cc:460] Webserver started at http://127.0.0.1:37347/ using document root <none> and password file <none>
I20260926 21:35:28.822002 21754 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:28.822148 21754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:28.822503 21754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:28.825408 21754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "619c97b59a5a4a1eb799ca33d6f636f7"
format_stamp: "Formatted at 2026-09-26 21:35:28 on dist-test-slave-87bt"
server_key: "45f87fd2017a0d28779ec4595baa2b6c"
server_key_iv: "6dcaed47486259c93d114d4722daa8dd"
server_key_version: "encryptionkey@0"
I20260926 21:35:28.826220 21754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "619c97b59a5a4a1eb799ca33d6f636f7"
format_stamp: "Formatted at 2026-09-26 21:35:28 on dist-test-slave-87bt"
server_key: "45f87fd2017a0d28779ec4595baa2b6c"
server_key_iv: "6dcaed47486259c93d114d4722daa8dd"
server_key_version: "encryptionkey@0"
I20260926 21:35:28.832290 21754 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260926 21:35:28.837615 21769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:28.839488 21754 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 21:35:28.839711 21754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "619c97b59a5a4a1eb799ca33d6f636f7"
format_stamp: "Formatted at 2026-09-26 21:35:28 on dist-test-slave-87bt"
server_key: "45f87fd2017a0d28779ec4595baa2b6c"
server_key_iv: "6dcaed47486259c93d114d4722daa8dd"
server_key_version: "encryptionkey@0"
I20260926 21:35:28.839985 21754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:28.851845 21754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:28.855631 21754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:28.855906 21754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:28.873893 21754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44141
I20260926 21:35:28.873916 21820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44141 every 8 connection(s)
I20260926 21:35:28.875360 21754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:28.876751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21754
I20260926 21:35:28.876936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:28.877367 388 external_mini_cluster.cc:1428] Setting key 6fd255f82b5027025db4ee7371800146
I20260926 21:35:28.879667 21821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:35:28 dist-test-slave-87bt krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:35:28.892752 21821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7: Bootstrap starting.
I20260926 21:35:28.896857 21821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:28.898113 21821 log.cc:824] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:28.901551 21821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7: No bootstrap required, opened a new log
I20260926 21:35:28.902019 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:28.879208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:57677 (local address 127.0.0.1:44141)
0926 21:35:28.879998 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:28.880012 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:28.880061 (+ 49us) server_negotiation.cc:407] Connection header received
0926 21:35:28.882323 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:28.882350 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:28.882898 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:28.883354 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:28.884703 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:28.886760 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:28.888007 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:28.889029 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:28.889218 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:28.893432 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:28.893444 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:28.893463 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:28.893505 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:28.897510 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:28.898295 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:28.898306 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:28.898314 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:28.898545 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:28.899078 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:28.899081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:28.899083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:28.899672 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:28.899966 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:28.900372 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 21:35:28.900712 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":325,"threads_started":1}
I20260926 21:35:28.905874 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:28.910219 21821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619c97b59a5a4a1eb799ca33d6f636f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } }
I20260926 21:35:28.910775 21821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:28.910905 21821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c97b59a5a4a1eb799ca33d6f636f7, State: Initialized, Role: FOLLOWER
I20260926 21:35:28.911602 21821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619c97b59a5a4a1eb799ca33d6f636f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } }
I20260926 21:35:28.911891 21821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:28.912051 21821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:28.912241 21821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:28.915189 21821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619c97b59a5a4a1eb799ca33d6f636f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } }
I20260926 21:35:28.915747 21821 leader_election.cc:302] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619c97b59a5a4a1eb799ca33d6f636f7; no voters:
I20260926 21:35:28.916392 21821 leader_election.cc:288] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:28.916512 21831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:28.918728 21821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:28.919440 21831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [term 1 LEADER]: Becoming Leader. State: Replica: 619c97b59a5a4a1eb799ca33d6f636f7, State: Running, Role: LEADER
I20260926 21:35:28.920467 21831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619c97b59a5a4a1eb799ca33d6f636f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } }
I20260926 21:35:28.923503 21833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619c97b59a5a4a1eb799ca33d6f636f7. Latest consensus state: current_term: 1 leader_uuid: "619c97b59a5a4a1eb799ca33d6f636f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619c97b59a5a4a1eb799ca33d6f636f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } } }
I20260926 21:35:28.923934 21833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:28.928812 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:28.910132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:56953 (local address 127.0.0.1:44141)
0926 21:35:28.910322 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:28.910330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:28.910713 (+ 383us) server_negotiation.cc:407] Connection header received
0926 21:35:28.910919 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:28.910923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:28.910982 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:28.911095 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:28.912932 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:28.913747 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:28.915538 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:28.916057 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:28.916187 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:28.919347 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:28.919357 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:28.919364 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:28.919410 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:28.923507 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:28.924765 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:28.924773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:28.924780 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:28.925040 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:28.925760 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:28.925767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:28.925773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:28.926285 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:28.927881 (+ 1596us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:28.928128 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 21:35:28.928381 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:35:28.932936 21832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619c97b59a5a4a1eb799ca33d6f636f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619c97b59a5a4a1eb799ca33d6f636f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44141 } } }
I20260926 21:35:28.933262 21832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:28.933270 21839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:28.939133 21839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:28.948975 21839 catalog_manager.cc:1347] Generated new cluster ID: 018349cf7a534f39b9e09472786db60c
I20260926 21:35:28.949124 21839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:28.962574 21839 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:28.964238 21839 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:28.976400 21839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619c97b59a5a4a1eb799ca33d6f636f7: Generated new TSK 0
I20260926 21:35:28.977314 21839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:28.989750 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21754
I20260926 21:35:28.990881 21816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:29.082789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21754
W20260926 21:35:29.100365 21739 connection.cc:535] client connection to 127.0.0.1:44141 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44141): Connection reset by peer (error 104)
2026-09-26T21:35:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:29 dist-test-slave-87bt krb5kdc[21854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:29 dist-test-slave-87bt krb5kdc[21854](info): set up 2 sockets
Sep 26 21:35:29 dist-test-slave-87bt krb5kdc[21854](info): commencing operation
krb5kdc: starting...
W20260926 21:35:30.186896 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:30 dist-test-slave-87bt krb5kdc[21854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:30.449110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:30.780817 21867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:30.781124 21867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:30.781284 21867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:30.788515 21867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:30.788632 21867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:30.788715 21867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:30.788800 21867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:30.788923 21867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:30.798910 21867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:30.800238 21867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:30.801556 21867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:30.813915 21872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:30.813968 21873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:30.813943 21875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:30.814818 21867 server_base.cc:1034] running on GCE node
I20260926 21:35:30.815616 21867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:30.817044 21867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:30.818269 21867 hybrid_clock.cc:648] HybridClock initialized: now 1790458530818219 us; error 45 us; skew 500 ppm
I20260926 21:35:30.818656 21867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:30 dist-test-slave-87bt krb5kdc[21854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:30.821751 21867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:30.823307 21867 webserver.cc:460] Webserver started at http://127.0.0.1:40861/ using document root <none> and password file <none>
I20260926 21:35:30.824011 21867 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:30.824153 21867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:30.824450 21867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:30.827575 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "bdf4c346d0af487ea94c974dc96a56a2"
format_stamp: "Formatted at 2026-09-26 21:35:30 on dist-test-slave-87bt"
server_key: "7db5073a69498119038c45e1632554b2"
server_key_iv: "97e322862986a0072868411062ad69a7"
server_key_version: "encryptionkey@0"
I20260926 21:35:30.828368 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "bdf4c346d0af487ea94c974dc96a56a2"
format_stamp: "Formatted at 2026-09-26 21:35:30 on dist-test-slave-87bt"
server_key: "7db5073a69498119038c45e1632554b2"
server_key_iv: "97e322862986a0072868411062ad69a7"
server_key_version: "encryptionkey@0"
I20260926 21:35:30.835390 21867 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260926 21:35:30.840667 21882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:30.842176 21867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:35:30.842388 21867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "bdf4c346d0af487ea94c974dc96a56a2"
format_stamp: "Formatted at 2026-09-26 21:35:30 on dist-test-slave-87bt"
server_key: "7db5073a69498119038c45e1632554b2"
server_key_iv: "97e322862986a0072868411062ad69a7"
server_key_version: "encryptionkey@0"
I20260926 21:35:30.842653 21867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:30.860255 21867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:30.864799 21867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:30.865053 21867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:30.883955 21867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41923
I20260926 21:35:30.883976 21933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41923 every 8 connection(s)
I20260926 21:35:30.885528 21867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:30.889441 21934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:30.893615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21867
I20260926 21:35:30.893795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:30.894333 388 external_mini_cluster.cc:1428] Setting key 579f2d104363ab3329a66fcb490f7e98
I20260926 21:35:30.902696 21934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2: Bootstrap starting.
I20260926 21:35:30.906399 21934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:30.907653 21934 log.cc:824] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:30.908182 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:30.896202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:44475 (local address 127.0.0.1:41923)
0926 21:35:30.896959 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:30.896971 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:30.897012 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:35:30.898675 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:30.898697 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:30.899151 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:30.899491 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:30.900194 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:30.901521 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:30.903397 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:30.904097 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:30.904235 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:30.905596 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:30.905604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:30.905628 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:30.906219 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:30.906296 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:30.906324 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:30.906533 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:30.906883 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 21:35:30.907085 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":338,"threads_started":1}
I20260926 21:35:30.911162 21934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2: No bootstrap required, opened a new log
I20260926 21:35:30.912551 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:30.919876 21934 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf4c346d0af487ea94c974dc96a56a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } }
I20260926 21:35:30.920408 21934 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:30.920536 21934 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf4c346d0af487ea94c974dc96a56a2, State: Initialized, Role: FOLLOWER
I20260926 21:35:30.921198 21934 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf4c346d0af487ea94c974dc96a56a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } }
I20260926 21:35:30.921463 21934 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:30.921610 21934 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:30.921782 21934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:30.924260 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:30.916982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:47701 (local address 127.0.0.1:41923)
0926 21:35:30.917178 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:30.917184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:30.917566 (+ 382us) server_negotiation.cc:407] Connection header received
0926 21:35:30.917730 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:30.917736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:30.917825 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:30.917956 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:30.918453 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:30.919625 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:30.921475 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:30.922216 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:30.922352 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:30.923553 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:30.923560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:30.923565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:30.923587 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:30.923635 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:30.923657 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:30.923775 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:30.923903 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 21:35:30.923959 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:35:30.924727 21934 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf4c346d0af487ea94c974dc96a56a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } }
I20260926 21:35:30.925335 21934 leader_election.cc:302] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf4c346d0af487ea94c974dc96a56a2; no voters:
I20260926 21:35:30.926029 21934 leader_election.cc:288] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:30.926134 21944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:30.926887 21944 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [term 1 LEADER]: Becoming Leader. State: Replica: bdf4c346d0af487ea94c974dc96a56a2, State: Running, Role: LEADER
I20260926 21:35:30.927448 21944 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf4c346d0af487ea94c974dc96a56a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } }
I20260926 21:35:30.928380 21934 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:30.930509 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf4c346d0af487ea94c974dc96a56a2. Latest consensus state: current_term: 1 leader_uuid: "bdf4c346d0af487ea94c974dc96a56a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf4c346d0af487ea94c974dc96a56a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } } }
I20260926 21:35:30.930884 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:30.931659 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf4c346d0af487ea94c974dc96a56a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf4c346d0af487ea94c974dc96a56a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } } }
I20260926 21:35:30.932008 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:30.932934 21949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:30.938346 21949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:30.948192 21949 catalog_manager.cc:1347] Generated new cluster ID: bfe44ea413ea4666b1427d8773765b42
I20260926 21:35:30.948328 21949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:30.976889 21949 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:30.978250 21949 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:30.991060 21949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf4c346d0af487ea94c974dc96a56a2: Generated new TSK 0
I20260926 21:35:30.991884 21949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:31.000497 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21867
W20260926 21:35:31.001729 21929 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.94:44475
W20260926 21:35:31.002542 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21867: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:31.002697 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21867
W20260926 21:35:31.019629 21852 connection.cc:535] client connection to 127.0.0.1:41923 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41923): Connection reset by peer (error 104)
2026-09-26T21:35:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:31 dist-test-slave-87bt krb5kdc[21965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:31 dist-test-slave-87bt krb5kdc[21965](info): set up 2 sockets
Sep 26 21:35:31 dist-test-slave-87bt krb5kdc[21965](info): commencing operation
krb5kdc: starting...
W20260926 21:35:32.123167 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:32 dist-test-slave-87bt krb5kdc[21965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:35:32.385667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:32.716512 21978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:32.716807 21978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:32.716952 21978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:32.724319 21978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:32.724439 21978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:32.724586 21978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:32.724689 21978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:32.724781 21978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:32.735097 21978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:32.736471 21978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:32.737799 21978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:32.750161 21983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:32.750120 21986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:32.750083 21984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:32.750675 21978 server_base.cc:1034] running on GCE node
I20260926 21:35:32.751782 21978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:32.753191 21978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:32.754458 21978 hybrid_clock.cc:648] HybridClock initialized: now 1790458532754378 us; error 59 us; skew 500 ppm
I20260926 21:35:32.754822 21978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:32 dist-test-slave-87bt krb5kdc[21965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:32.765132 21978 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:35:32.766711 21978 webserver.cc:460] Webserver started at http://127.0.0.1:34969/ using document root <none> and password file <none>
I20260926 21:35:32.767419 21978 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:32.767562 21978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:32.767871 21978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:32.770987 21978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "8f6ec80d0a814e6a931cef5eb6b18b99"
format_stamp: "Formatted at 2026-09-26 21:35:32 on dist-test-slave-87bt"
server_key: "615c05221d72493d6f55651f26d37ad7"
server_key_iv: "31aaac90894fe4d4e58763d830e9fb17"
server_key_version: "encryptionkey@0"
I20260926 21:35:32.771780 21978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f6ec80d0a814e6a931cef5eb6b18b99"
format_stamp: "Formatted at 2026-09-26 21:35:32 on dist-test-slave-87bt"
server_key: "615c05221d72493d6f55651f26d37ad7"
server_key_iv: "31aaac90894fe4d4e58763d830e9fb17"
server_key_version: "encryptionkey@0"
I20260926 21:35:32.776907 21978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:35:32.780434 21993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:32.781759 21978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:35:32.781948 21978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "8f6ec80d0a814e6a931cef5eb6b18b99"
format_stamp: "Formatted at 2026-09-26 21:35:32 on dist-test-slave-87bt"
server_key: "615c05221d72493d6f55651f26d37ad7"
server_key_iv: "31aaac90894fe4d4e58763d830e9fb17"
server_key_version: "encryptionkey@0"
I20260926 21:35:32.782148 21978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:32.787761 21978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:32.791714 21978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:32.792001 21978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:32.810263 21978 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44591
I20260926 21:35:32.810319 22044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44591 every 8 connection(s)
I20260926 21:35:32.811707 21978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:32.815413 22045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:32.820536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 21978
I20260926 21:35:32.820732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:32.821152 388 external_mini_cluster.cc:1428] Setting key 4b762f0837586317457f4f350cf950fd
I20260926 21:35:32.827785 22045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99: Bootstrap starting.
I20260926 21:35:32.830453 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:32.823084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:39455 (local address 127.0.0.1:44591)
0926 21:35:32.823917 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:32.823933 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:32.823992 (+ 59us) server_negotiation.cc:407] Connection header received
0926 21:35:32.826529 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:32.826560 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:32.827017 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:32.827495 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:32.828507 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:32.828516 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:32.828534 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:32.828550 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:32.828618 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:32.828643 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:32.828846 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:32.828975 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 21:35:32.829168 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":334,"threads_started":1}
I20260926 21:35:32.832015 22045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:32.833554 22045 log.cc:824] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:32.838160 22045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99: No bootstrap required, opened a new log
I20260926 21:35:32.838214 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:32.845639 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:32.842940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:47003 (local address 127.0.0.1:44591)
0926 21:35:32.843199 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:32.843212 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:32.843749 (+ 537us) server_negotiation.cc:407] Connection header received
0926 21:35:32.843885 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:32.843891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:32.843964 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:32.844083 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:32.844969 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:32.844978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:32.844986 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:32.845007 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:32.845053 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:32.845073 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:32.845211 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:32.845256 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 21:35:32.845319 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:35:32.846881 22045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ec80d0a814e6a931cef5eb6b18b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44591 } }
I20260926 21:35:32.847342 22045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:32.847456 22045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6ec80d0a814e6a931cef5eb6b18b99, State: Initialized, Role: FOLLOWER
I20260926 21:35:32.848102 22045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ec80d0a814e6a931cef5eb6b18b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44591 } }
I20260926 21:35:32.848358 22045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:32.848497 22045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:32.848682 22045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:32.851212 22045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ec80d0a814e6a931cef5eb6b18b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44591 } }
I20260926 21:35:32.851734 22045 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f6ec80d0a814e6a931cef5eb6b18b99; no voters:
I20260926 21:35:32.852324 22045 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:32.852476 22055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:32.853140 22055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [term 1 LEADER]: Becoming Leader. State: Replica: 8f6ec80d0a814e6a931cef5eb6b18b99, State: Running, Role: LEADER
I20260926 21:35:32.853681 22055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ec80d0a814e6a931cef5eb6b18b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44591 } }
I20260926 21:35:32.854554 22045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:32.856953 22056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f6ec80d0a814e6a931cef5eb6b18b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ec80d0a814e6a931cef5eb6b18b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44591 } } }
I20260926 21:35:32.857051 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6ec80d0a814e6a931cef5eb6b18b99. Latest consensus state: current_term: 1 leader_uuid: "8f6ec80d0a814e6a931cef5eb6b18b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ec80d0a814e6a931cef5eb6b18b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44591 } } }
I20260926 21:35:32.857354 22056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:32.857438 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:32.859934 22060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:32.865132 22060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:32.873519 22060 catalog_manager.cc:1347] Generated new cluster ID: a32153721a964f1ab79ddd22e0767c42
I20260926 21:35:32.873638 22060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:32.908721 22060 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:32.909168 22060 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:32.915334 22060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6ec80d0a814e6a931cef5eb6b18b99: Generated new TSK 0
I20260926 21:35:32.916088 22060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:32.928177 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21978
W20260926 21:35:32.929901 22040 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.94:39455
W20260926 21:35:32.930655 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 21978: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:32.930905 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 21978
2026-09-26T21:35:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:33 dist-test-slave-87bt krb5kdc[22076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:33 dist-test-slave-87bt krb5kdc[22076](info): set up 2 sockets
Sep 26 21:35:33 dist-test-slave-87bt krb5kdc[22076](info): commencing operation
krb5kdc: starting...
W20260926 21:35:34.034940 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:34 dist-test-slave-87bt krb5kdc[22076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:34.310773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42471
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:34.647405 22089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:34.647704 22089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:34.647908 22089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:34.655202 22089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:34.655318 22089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:34.655444 22089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:34.655555 22089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:34.655642 22089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:34.665674 22089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:34.667037 22089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:34.668540 22089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:34.680845 22094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:34.680873 22097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:34.680912 22095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:34.681787 22089 server_base.cc:1034] running on GCE node
I20260926 21:35:34.682711 22089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:34.684252 22089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:34.685484 22089 hybrid_clock.cc:648] HybridClock initialized: now 1790458534685439 us; error 36 us; skew 500 ppm
I20260926 21:35:34.685847 22089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:34 dist-test-slave-87bt krb5kdc[22076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:34.689253 22089 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:34.691047 22089 webserver.cc:460] Webserver started at http://127.0.97.62:36077/ using document root <none> and password file <none>
I20260926 21:35:34.691779 22089 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:34.691938 22089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:34.692253 22089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:34.695426 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "c73f6a4b39564710ad9db31b87a02492"
format_stamp: "Formatted at 2026-09-26 21:35:34 on dist-test-slave-87bt"
server_key: "0e55fa0047d284d12abfa00a67aad8bd"
server_key_iv: "1a18d5d106abf32ed56c81fd6e943026"
server_key_version: "encryptionkey@0"
I20260926 21:35:34.696200 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "c73f6a4b39564710ad9db31b87a02492"
format_stamp: "Formatted at 2026-09-26 21:35:34 on dist-test-slave-87bt"
server_key: "0e55fa0047d284d12abfa00a67aad8bd"
server_key_iv: "1a18d5d106abf32ed56c81fd6e943026"
server_key_version: "encryptionkey@0"
I20260926 21:35:34.701341 22089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:35:34.704788 22104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:34.706122 22089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:35:34.706315 22089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "c73f6a4b39564710ad9db31b87a02492"
format_stamp: "Formatted at 2026-09-26 21:35:34 on dist-test-slave-87bt"
server_key: "0e55fa0047d284d12abfa00a67aad8bd"
server_key_iv: "1a18d5d106abf32ed56c81fd6e943026"
server_key_version: "encryptionkey@0"
I20260926 21:35:34.706578 22089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:34.726579 22089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:34.731248 22089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:34.731559 22089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:34.750537 22089 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42471
I20260926 21:35:34.750590 22155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42471 every 8 connection(s)
I20260926 21:35:34.751967 22089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:34.755686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22089
I20260926 21:35:34.755887 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:34.756014 22156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:34.756377 388 external_mini_cluster.cc:1428] Setting key 247fd02a6df8aefb00958a204d80f297
I20260926 21:35:34.770856 22156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492: Bootstrap starting.
Sep 26 21:35:34 dist-test-slave-87bt krb5kdc[22076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:34.774933 22156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:34.776399 22156 log.cc:824] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:34.780057 22156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492: No bootstrap required, opened a new log
I20260926 21:35:34.781502 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:34.758249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52095 (local address 127.0.97.62:42471)
0926 21:35:34.759154 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:34.759175 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:34.759230 (+ 55us) server_negotiation.cc:407] Connection header received
0926 21:35:34.761160 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:34.761190 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:34.761735 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:34.762211 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:34.763524 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:34.765511 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:34.766939 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:34.768092 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:34.768283 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:34.772369 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:34.772383 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:34.772406 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:34.772463 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:34.776607 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:34.777605 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:34.777612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:34.777621 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:34.777862 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:34.778312 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:34.778317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:34.778320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:34.779023 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:34.779348 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:34.779735 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 21:35:34.780104 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":394,"threads_started":1}
I20260926 21:35:34.785475 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:34.789031 22156 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73f6a4b39564710ad9db31b87a02492" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42471 } }
I20260926 21:35:34.789446 22156 raft_consensus.cc:374] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:34.789594 22156 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73f6a4b39564710ad9db31b87a02492, State: Initialized, Role: FOLLOWER
I20260926 21:35:34.790234 22156 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73f6a4b39564710ad9db31b87a02492" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42471 } }
I20260926 21:35:34.790541 22156 raft_consensus.cc:388] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:34.790709 22156 raft_consensus.cc:482] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:34.790901 22156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:34.793777 22156 raft_consensus.cc:504] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73f6a4b39564710ad9db31b87a02492" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42471 } }
I20260926 21:35:34.794317 22156 leader_election.cc:302] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c73f6a4b39564710ad9db31b87a02492; no voters:
I20260926 21:35:34.794968 22156 leader_election.cc:288] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:34.795135 22166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:34.796864 22166 raft_consensus.cc:686] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [term 1 LEADER]: Becoming Leader. State: Replica: c73f6a4b39564710ad9db31b87a02492, State: Running, Role: LEADER
I20260926 21:35:34.797482 22166 consensus_queue.cc:237] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73f6a4b39564710ad9db31b87a02492" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42471 } }
I20260926 21:35:34.797745 22156 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:34.804275 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c73f6a4b39564710ad9db31b87a02492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73f6a4b39564710ad9db31b87a02492" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42471 } } }
I20260926 21:35:34.804672 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:34.805208 22168 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c73f6a4b39564710ad9db31b87a02492. Latest consensus state: current_term: 1 leader_uuid: "c73f6a4b39564710ad9db31b87a02492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73f6a4b39564710ad9db31b87a02492" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42471 } } }
I20260926 21:35:34.805574 22168 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:34.807541 22175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:34.809631 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:34.789691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54225 (local address 127.0.97.62:42471)
0926 21:35:34.789925 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:34.789932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:34.790618 (+ 686us) server_negotiation.cc:407] Connection header received
0926 21:35:34.790801 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:34.790806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:34.790898 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:34.791016 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:34.792335 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:34.793490 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:34.795061 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:34.795820 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:34.796575 (+ 755us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:34.799119 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:34.799129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:34.799137 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:34.799189 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:34.804098 (+ 4909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:34.805494 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:34.805503 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:34.805509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:34.805696 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:34.806410 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:34.806461 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:34.806467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:34.807081 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:34.808599 (+ 1518us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:34.808862 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 21:35:34.809117 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:35:34.817626 22175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:34.827716 22175 catalog_manager.cc:1347] Generated new cluster ID: 2ac1c3d37cdc4c3c8df2777568193400
I20260926 21:35:34.827852 22175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:34.840265 22175 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:34.841583 22175 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:34.850056 22175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c73f6a4b39564710ad9db31b87a02492: Generated new TSK 0
I20260926 21:35:34.851172 22175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:34.864495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22089
I20260926 21:35:34.865607 22151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:34.955972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22089
W20260926 21:35:34.975893 22074 connection.cc:535] client connection to 127.0.97.62:42471 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42471): Connection reset by peer (error 104)
2026-09-26T21:35:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2025 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:35 dist-test-slave-87bt krb5kdc[22188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:35 dist-test-slave-87bt krb5kdc[22188](info): set up 2 sockets
Sep 26 21:35:35 dist-test-slave-87bt krb5kdc[22188](info): commencing operation
krb5kdc: starting...
W20260926 21:35:36.062999 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:36 dist-test-slave-87bt krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:36.326607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32857
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:36.663790 22201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:36.664100 22201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:36.664278 22201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:36.671885 22201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:36.672012 22201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:36.672094 22201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:36.672212 22201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:36.672298 22201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:36.682474 22201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:36.683861 22201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:36.685237 22201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:36.697661 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:36.698213 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:36.698395 22206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:36.698935 22201 server_base.cc:1034] running on GCE node
I20260926 21:35:36.699864 22201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:36.701565 22201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:36.702808 22201 hybrid_clock.cc:648] HybridClock initialized: now 1790458536702752 us; error 44 us; skew 500 ppm
I20260926 21:35:36.703246 22201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:36 dist-test-slave-87bt krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:36.706851 22201 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:36.708681 22201 webserver.cc:460] Webserver started at http://127.0.97.62:39225/ using document root <none> and password file <none>
I20260926 21:35:36.709478 22201 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:36.709651 22201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:36.710000 22201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:36.713387 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "0e4803efa14f4a78ad90e91aca973475"
format_stamp: "Formatted at 2026-09-26 21:35:36 on dist-test-slave-87bt"
server_key: "1d38a8308a1b65083bf91ed44d14ec18"
server_key_iv: "a05dedf18f8981e66cc5b25f60d81c62"
server_key_version: "encryptionkey@0"
I20260926 21:35:36.714263 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "0e4803efa14f4a78ad90e91aca973475"
format_stamp: "Formatted at 2026-09-26 21:35:36 on dist-test-slave-87bt"
server_key: "1d38a8308a1b65083bf91ed44d14ec18"
server_key_iv: "a05dedf18f8981e66cc5b25f60d81c62"
server_key_version: "encryptionkey@0"
I20260926 21:35:36.720279 22201 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260926 21:35:36.724643 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:36.726228 22201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:35:36.726547 22201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "0e4803efa14f4a78ad90e91aca973475"
format_stamp: "Formatted at 2026-09-26 21:35:36 on dist-test-slave-87bt"
server_key: "1d38a8308a1b65083bf91ed44d14ec18"
server_key_iv: "a05dedf18f8981e66cc5b25f60d81c62"
server_key_version: "encryptionkey@0"
I20260926 21:35:36.726790 22201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:36.754758 22201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:36.759570 22201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:36.759884 22201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:36.779434 22201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32857
I20260926 21:35:36.779474 22267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32857 every 8 connection(s)
I20260926 21:35:36.780871 22201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:36.782548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22201
I20260926 21:35:36.782725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:36.783270 388 external_mini_cluster.cc:1428] Setting key 3712821aa0314f2211d334fe673ec632
I20260926 21:35:36.785648 22268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:36.797197 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:36.785112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43101 (local address 127.0.97.62:32857)
0926 21:35:36.785956 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:36.785969 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:36.786011 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:35:36.787765 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:36.787788 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:36.788216 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:36.788581 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:36.789165 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:36.790612 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:36.792515 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:36.793196 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:36.793330 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:36.794713 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:36.794721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:36.794736 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:36.795349 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:36.795438 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:36.795469 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:36.795601 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:36.795933 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 21:35:36.796148 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":376,"threads_started":1}
I20260926 21:35:36.798767 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475: Bootstrap starting.
I20260926 21:35:36.801347 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:36.802598 22268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:36.803692 22268 log.cc:824] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:36.807407 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475: No bootstrap required, opened a new log
I20260926 21:35:36.812942 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:36.805899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47545 (local address 127.0.97.62:32857)
0926 21:35:36.806128 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:36.806134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:36.806554 (+ 420us) server_negotiation.cc:407] Connection header received
0926 21:35:36.806717 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:36.806723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:36.806808 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:36.806936 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:36.807449 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:36.808612 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:36.809976 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:36.810751 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:36.810904 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:36.812077 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:36.812084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:36.812089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:36.812113 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:36.812166 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:36.812189 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:36.812317 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:36.812490 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 21:35:36.812552 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:35:36.817080 22268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4803efa14f4a78ad90e91aca973475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32857 } }
I20260926 21:35:36.817499 22268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:36.817584 22268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4803efa14f4a78ad90e91aca973475, State: Initialized, Role: FOLLOWER
I20260926 21:35:36.818228 22268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4803efa14f4a78ad90e91aca973475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32857 } }
I20260926 21:35:36.818528 22268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:36.818662 22268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:36.818813 22268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:36.821082 22268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4803efa14f4a78ad90e91aca973475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32857 } }
I20260926 21:35:36.821692 22268 leader_election.cc:302] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e4803efa14f4a78ad90e91aca973475; no voters:
I20260926 21:35:36.822352 22268 leader_election.cc:288] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:36.822613 22278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:36.823422 22278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4803efa14f4a78ad90e91aca973475, State: Running, Role: LEADER
I20260926 21:35:36.824015 22278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4803efa14f4a78ad90e91aca973475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32857 } }
I20260926 21:35:36.825055 22268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:36.827775 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e4803efa14f4a78ad90e91aca973475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4803efa14f4a78ad90e91aca973475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32857 } } }
I20260926 21:35:36.828297 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:36.827749 22280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e4803efa14f4a78ad90e91aca973475. Latest consensus state: current_term: 1 leader_uuid: "0e4803efa14f4a78ad90e91aca973475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4803efa14f4a78ad90e91aca973475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32857 } } }
I20260926 21:35:36.828684 22280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:36.837843 22286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:36.846233 22286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:36.858191 22286 catalog_manager.cc:1347] Generated new cluster ID: 7a47abb6303e45e09b00d85e2b9ea5fa
I20260926 21:35:36.858337 22286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:36.868448 22286 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:36.869867 22286 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:36.880431 22286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e4803efa14f4a78ad90e91aca973475: Generated new TSK 0
I20260926 21:35:36.881439 22286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:36.888535 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22201
W20260926 21:35:36.889741 22263 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43101
W20260926 21:35:36.890595 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22201: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:36.890763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22201
W20260926 21:35:36.907913 22186 connection.cc:535] client connection to 127.0.97.62:32857 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:32857): Connection reset by peer (error 104)
2026-09-26T21:35:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:36 dist-test-slave-87bt krb5kdc[22299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:36 dist-test-slave-87bt krb5kdc[22299](info): set up 2 sockets
Sep 26 21:35:36 dist-test-slave-87bt krb5kdc[22299](info): commencing operation
krb5kdc: starting...
W20260926 21:35:38.006947 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:38 dist-test-slave-87bt krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:38.269434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38995
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:38.604692 22312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:38.604998 22312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:38.605198 22312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:38.612506 22312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:38.612627 22312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:38.612746 22312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:38.612857 22312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:38.612944 22312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:38.623061 22312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:38.624476 22312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:38.625938 22312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:38.638908 22320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:38.638944 22318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:38.639422 22317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:38.639806 22312 server_base.cc:1034] running on GCE node
I20260926 21:35:38.640786 22312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:38.642663 22312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:38.643981 22312 hybrid_clock.cc:648] HybridClock initialized: now 1790458538643933 us; error 74 us; skew 500 ppm
I20260926 21:35:38.644470 22312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:38 dist-test-slave-87bt krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:38.651845 22312 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:38.653380 22312 webserver.cc:460] Webserver started at http://127.0.97.62:45963/ using document root <none> and password file <none>
I20260926 21:35:38.654063 22312 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:38.654161 22312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:38.654469 22312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:38.657459 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "c9b9720c83e74736bd9e3746605fb54b"
format_stamp: "Formatted at 2026-09-26 21:35:38 on dist-test-slave-87bt"
server_key: "ecda67ecd8d04d9f85b2bb29c8de85c4"
server_key_iv: "6f09ebd6edb8b94ba44a187e2b79c6c8"
server_key_version: "encryptionkey@0"
I20260926 21:35:38.658175 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "c9b9720c83e74736bd9e3746605fb54b"
format_stamp: "Formatted at 2026-09-26 21:35:38 on dist-test-slave-87bt"
server_key: "ecda67ecd8d04d9f85b2bb29c8de85c4"
server_key_iv: "6f09ebd6edb8b94ba44a187e2b79c6c8"
server_key_version: "encryptionkey@0"
I20260926 21:35:38.663455 22312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:35:38.666864 22327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:38.668161 22312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:35:38.668303 22312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "c9b9720c83e74736bd9e3746605fb54b"
format_stamp: "Formatted at 2026-09-26 21:35:38 on dist-test-slave-87bt"
server_key: "ecda67ecd8d04d9f85b2bb29c8de85c4"
server_key_iv: "6f09ebd6edb8b94ba44a187e2b79c6c8"
server_key_version: "encryptionkey@0"
I20260926 21:35:38.668476 22312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:38.674141 22312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:38.678020 22312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:38.678252 22312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:38.696986 22312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38995
I20260926 21:35:38.697029 22378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38995 every 8 connection(s)
I20260926 21:35:38.698470 22312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:38.702131 22379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:38.704165 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22312
I20260926 21:35:38.704336 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:38.704867 388 external_mini_cluster.cc:1428] Setting key c6f04dc6f2fa67b5af989103e2f4afee
I20260926 21:35:38.712178 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:38.706640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53091 (local address 127.0.97.62:38995)
0926 21:35:38.707432 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:38.707443 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:38.707486 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:35:38.709014 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:38.709036 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:38.709371 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:38.709681 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:38.710702 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:38.710708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:38.710721 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:38.710734 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:38.710790 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:38.710814 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:38.710991 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:38.711085 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 21:35:38.711281 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":388,"threads_started":1}
I20260926 21:35:38.715312 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b: Bootstrap starting.
I20260926 21:35:38.715643 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:38.719159 22379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:38.720597 22379 log.cc:824] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:38.723345 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:38.720926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33425 (local address 127.0.97.62:38995)
0926 21:35:38.721077 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:38.721081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:38.721469 (+ 388us) server_negotiation.cc:407] Connection header received
0926 21:35:38.721630 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:38.721638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:38.721706 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:38.721840 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:38.722728 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:38.722732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:38.722736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:38.722751 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:38.722787 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:38.722802 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:38.722902 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:38.723009 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 21:35:38.723053 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 21:35:38.724150 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b: No bootstrap required, opened a new log
I20260926 21:35:38.731542 22379 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b9720c83e74736bd9e3746605fb54b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } }
I20260926 21:35:38.731971 22379 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:38.732079 22379 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b9720c83e74736bd9e3746605fb54b, State: Initialized, Role: FOLLOWER
I20260926 21:35:38.732676 22379 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b9720c83e74736bd9e3746605fb54b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } }
I20260926 21:35:38.732935 22379 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:38.733074 22379 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:38.733261 22379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:38.735561 22379 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b9720c83e74736bd9e3746605fb54b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } }
I20260926 21:35:38.736084 22379 leader_election.cc:302] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9b9720c83e74736bd9e3746605fb54b; no voters:
I20260926 21:35:38.736678 22379 leader_election.cc:288] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:38.736855 22389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:38.738610 22389 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [term 1 LEADER]: Becoming Leader. State: Replica: c9b9720c83e74736bd9e3746605fb54b, State: Running, Role: LEADER
I20260926 21:35:38.738922 22379 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:38.739239 22389 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b9720c83e74736bd9e3746605fb54b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } }
I20260926 21:35:38.742633 22391 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9b9720c83e74736bd9e3746605fb54b. Latest consensus state: current_term: 1 leader_uuid: "c9b9720c83e74736bd9e3746605fb54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b9720c83e74736bd9e3746605fb54b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } } }
I20260926 21:35:38.743161 22391 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:38.742669 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9b9720c83e74736bd9e3746605fb54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b9720c83e74736bd9e3746605fb54b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } } }
I20260926 21:35:38.743762 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:38.747601 22398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:38.752599 22398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:38.760944 22398 catalog_manager.cc:1347] Generated new cluster ID: 9dcdcaec84aa44d79bf8c966c129ccd6
I20260926 21:35:38.761061 22398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:38.780515 22398 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:38.781030 22398 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:38.788327 22398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9b9720c83e74736bd9e3746605fb54b: Generated new TSK 0
I20260926 21:35:38.789453 22398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:38.797132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22312
W20260926 21:35:38.798381 22374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53091
W20260926 21:35:38.799162 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22312: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:38.799322 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22312
2026-09-26T21:35:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:38 dist-test-slave-87bt krb5kdc[22410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:38 dist-test-slave-87bt krb5kdc[22410](info): set up 2 sockets
Sep 26 21:35:38 dist-test-slave-87bt krb5kdc[22410](info): commencing operation
krb5kdc: starting...
W20260926 21:35:39.902890 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:40 dist-test-slave-87bt krb5kdc[22410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:40.165333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38461
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:40.505182 22423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:40.505467 22423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:40.505663 22423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:40.513164 22423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:40.513289 22423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:40.513402 22423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:40.513511 22423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:40.513593 22423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:40.523821 22423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:40.525233 22423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:40.526763 22423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:40.539746 22428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:40.539635 22429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:40.539664 22431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:40.540392 22423 server_base.cc:1034] running on GCE node
I20260926 21:35:40.541210 22423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:40.543072 22423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:40.544322 22423 hybrid_clock.cc:648] HybridClock initialized: now 1790458540544278 us; error 34 us; skew 500 ppm
I20260926 21:35:40.544654 22423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:40 dist-test-slave-87bt krb5kdc[22410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:40.547938 22423 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:40.549548 22423 webserver.cc:460] Webserver started at http://127.0.97.62:43695/ using document root <none> and password file <none>
I20260926 21:35:40.550300 22423 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:40.550525 22423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:40.550846 22423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:40.553902 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "171ca666910241bd9551476d83e68e27"
format_stamp: "Formatted at 2026-09-26 21:35:40 on dist-test-slave-87bt"
server_key: "f390c68b5a7d8dc9ccfd3a2d046ac873"
server_key_iv: "41ccd4b89e5ca6d09faa25d1977d0749"
server_key_version: "encryptionkey@0"
I20260926 21:35:40.554845 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "171ca666910241bd9551476d83e68e27"
format_stamp: "Formatted at 2026-09-26 21:35:40 on dist-test-slave-87bt"
server_key: "f390c68b5a7d8dc9ccfd3a2d046ac873"
server_key_iv: "41ccd4b89e5ca6d09faa25d1977d0749"
server_key_version: "encryptionkey@0"
I20260926 21:35:40.560154 22423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:35:40.563887 22438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:40.565279 22423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:35:40.565483 22423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "171ca666910241bd9551476d83e68e27"
format_stamp: "Formatted at 2026-09-26 21:35:40 on dist-test-slave-87bt"
server_key: "f390c68b5a7d8dc9ccfd3a2d046ac873"
server_key_iv: "41ccd4b89e5ca6d09faa25d1977d0749"
server_key_version: "encryptionkey@0"
I20260926 21:35:40.565692 22423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:40.581818 22423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:40.585772 22423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:40.586073 22423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:40.604468 22423 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38461
I20260926 21:35:40.604489 22489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38461 every 8 connection(s)
I20260926 21:35:40.605845 22423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:40.609532 22490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:40.611122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22423
I20260926 21:35:40.611325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:40.611820 388 external_mini_cluster.cc:1428] Setting key d9baeca17057a7e3e6d710072e40e259
I20260926 21:35:40.624223 22490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27: Bootstrap starting.
Sep 26 21:35:40 dist-test-slave-87bt krb5kdc[22410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:40.627533 22490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:40.629035 22490 log.cc:824] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:40.632849 22490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27: No bootstrap required, opened a new log
I20260926 21:35:40.636552 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:40.613716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35045 (local address 127.0.97.62:38461)
0926 21:35:40.614621 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:40.614635 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:40.614687 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:35:40.616416 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:40.616446 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:40.617029 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:40.617521 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:40.618827 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:40.620858 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:40.622218 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:40.623314 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:40.623520 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:40.627638 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:40.627657 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:40.627678 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:40.627730 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:40.631692 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:40.632458 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:40.632466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:40.632474 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:40.632696 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:40.633107 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:40.633114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:40.633117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:40.633864 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:40.634147 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:40.634544 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 21:35:40.635040 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":374,"threads_started":1}
I20260926 21:35:40.640765 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:40.642769 22490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171ca666910241bd9551476d83e68e27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38461 } }
I20260926 21:35:40.643211 22490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:40.643338 22490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171ca666910241bd9551476d83e68e27, State: Initialized, Role: FOLLOWER
I20260926 21:35:40.644084 22490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171ca666910241bd9551476d83e68e27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38461 } }
I20260926 21:35:40.644335 22490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:40.644524 22490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:40.644690 22490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:40.647517 22490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171ca666910241bd9551476d83e68e27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38461 } }
I20260926 21:35:40.648196 22490 leader_election.cc:302] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 171ca666910241bd9551476d83e68e27; no voters:
I20260926 21:35:40.648854 22490 leader_election.cc:288] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:40.649005 22500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:40.649680 22500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [term 1 LEADER]: Becoming Leader. State: Replica: 171ca666910241bd9551476d83e68e27, State: Running, Role: LEADER
I20260926 21:35:40.650339 22500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171ca666910241bd9551476d83e68e27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38461 } }
I20260926 21:35:40.652168 22500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 171ca666910241bd9551476d83e68e27. Latest consensus state: current_term: 1 leader_uuid: "171ca666910241bd9551476d83e68e27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171ca666910241bd9551476d83e68e27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38461 } } }
I20260926 21:35:40.652522 22500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:40.656068 22502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "171ca666910241bd9551476d83e68e27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171ca666910241bd9551476d83e68e27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38461 } } }
I20260926 21:35:40.656494 22502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:40.657676 22490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:40.657871 22504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:40.664481 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:40.645432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41559 (local address 127.0.97.62:38461)
0926 21:35:40.645636 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:40.645644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:40.646009 (+ 365us) server_negotiation.cc:407] Connection header received
0926 21:35:40.646215 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:40.646221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:40.646303 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:40.646424 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:40.647852 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:40.648973 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:40.650924 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:40.651896 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:40.652069 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:40.656384 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:40.656392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:40.656396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:40.656440 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:40.660423 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:40.661354 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:40.661360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:40.661364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:40.661545 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:40.662032 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:40.662038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:40.662042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:40.662510 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:40.662790 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:40.663892 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0926 21:35:40.664114 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:35:40.664661 22504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:40.676328 22504 catalog_manager.cc:1347] Generated new cluster ID: 45ddf715da61403a84a9842e26a8d548
I20260926 21:35:40.676477 22504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:40.697201 22504 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:40.698734 22504 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:40.714190 22504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 171ca666910241bd9551476d83e68e27: Generated new TSK 0
I20260926 21:35:40.715130 22504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:40.720991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22423
I20260926 21:35:40.722189 22485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:40.815112 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22423
2026-09-26T21:35:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:40 dist-test-slave-87bt krb5kdc[22522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:40 dist-test-slave-87bt krb5kdc[22522](info): set up 2 sockets
Sep 26 21:35:40 dist-test-slave-87bt krb5kdc[22522](info): commencing operation
krb5kdc: starting...
W20260926 21:35:41.914954 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:42 dist-test-slave-87bt krb5kdc[22522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:42.175887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33855
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:42.526525 22535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:42.526897 22535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:42.527103 22535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:42.534698 22535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:42.534829 22535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:42.534875 22535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:42.534929 22535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:42.534971 22535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:42.545047 22535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:42.546304 22535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:42.547762 22535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:42.560079 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:42.560029 22541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:42.560035 22543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:42.560732 22535 server_base.cc:1034] running on GCE node
I20260926 21:35:42.561734 22535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:42.563246 22535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:42.564488 22535 hybrid_clock.cc:648] HybridClock initialized: now 1790458542564442 us; error 41 us; skew 500 ppm
I20260926 21:35:42.564877 22535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:42 dist-test-slave-87bt krb5kdc[22522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:42.568082 22535 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:42.569598 22535 webserver.cc:460] Webserver started at http://127.0.97.62:38393/ using document root <none> and password file <none>
I20260926 21:35:42.570294 22535 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:42.570480 22535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:42.570797 22535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:42.573715 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "19cb5293d5c54b2bbf046944edd0d131"
format_stamp: "Formatted at 2026-09-26 21:35:42 on dist-test-slave-87bt"
server_key: "3248ec44e3cceb143d91d630bc518d4e"
server_key_iv: "1255e5f2e13d26828285ce2dbc88b0d9"
server_key_version: "encryptionkey@0"
I20260926 21:35:42.574533 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "19cb5293d5c54b2bbf046944edd0d131"
format_stamp: "Formatted at 2026-09-26 21:35:42 on dist-test-slave-87bt"
server_key: "3248ec44e3cceb143d91d630bc518d4e"
server_key_iv: "1255e5f2e13d26828285ce2dbc88b0d9"
server_key_version: "encryptionkey@0"
I20260926 21:35:42.579991 22535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 21:35:42.583323 22550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:42.584685 22535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:35:42.584894 22535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "19cb5293d5c54b2bbf046944edd0d131"
format_stamp: "Formatted at 2026-09-26 21:35:42 on dist-test-slave-87bt"
server_key: "3248ec44e3cceb143d91d630bc518d4e"
server_key_iv: "1255e5f2e13d26828285ce2dbc88b0d9"
server_key_version: "encryptionkey@0"
I20260926 21:35:42.585086 22535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:42.598157 22535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:42.602684 22535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:42.602981 22535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:42.621603 22535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33855
I20260926 21:35:42.621625 22601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33855 every 8 connection(s)
I20260926 21:35:42.623063 22535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:42.626863 22602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:42.631080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22535
I20260926 21:35:42.631264 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:42.631808 388 external_mini_cluster.cc:1428] Setting key 1862c66ec9e6c13e17bbfc1a967ba764
I20260926 21:35:42.639995 22602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131: Bootstrap starting.
I20260926 21:35:42.643273 22602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:42.644683 22602 log.cc:824] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:42.646466 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:42.633647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33389 (local address 127.0.97.62:33855)
0926 21:35:42.634612 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:42.634628 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:42.634676 (+ 48us) server_negotiation.cc:407] Connection header received
0926 21:35:42.636470 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:42.636496 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:42.636994 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:42.637472 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:42.638105 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:42.639570 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:42.641176 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:42.641957 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:42.642198 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:42.643573 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:42.643584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:42.643606 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:42.644383 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:42.644501 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:42.644534 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:42.644669 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:42.644917 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 21:35:42.645241 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":416,"threads_started":1}
I20260926 21:35:42.648080 22602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131: No bootstrap required, opened a new log
I20260926 21:35:42.650521 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:42.656533 22602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb5293d5c54b2bbf046944edd0d131" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33855 } }
I20260926 21:35:42.656991 22602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:42.657107 22602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19cb5293d5c54b2bbf046944edd0d131, State: Initialized, Role: FOLLOWER
I20260926 21:35:42.657791 22602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb5293d5c54b2bbf046944edd0d131" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33855 } }
I20260926 21:35:42.658090 22602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:42.658224 22602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:42.658391 22602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:42.661494 22602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb5293d5c54b2bbf046944edd0d131" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33855 } }
I20260926 21:35:42.661736 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:42.654955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49017 (local address 127.0.97.62:33855)
0926 21:35:42.655196 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:42.655203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:42.655474 (+ 271us) server_negotiation.cc:407] Connection header received
0926 21:35:42.655715 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:42.655720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:42.655784 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:42.655889 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:42.656450 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:42.657384 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:42.659238 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:42.659751 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:42.659919 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:42.661012 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:42.661017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:42.661021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:42.661036 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:42.661071 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:42.661085 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:42.661151 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:42.661307 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 21:35:42.661393 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 21:35:42.662137 22602 leader_election.cc:302] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19cb5293d5c54b2bbf046944edd0d131; no voters:
I20260926 21:35:42.662856 22602 leader_election.cc:288] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:42.662966 22612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:42.663697 22612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [term 1 LEADER]: Becoming Leader. State: Replica: 19cb5293d5c54b2bbf046944edd0d131, State: Running, Role: LEADER
I20260926 21:35:42.664306 22612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb5293d5c54b2bbf046944edd0d131" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33855 } }
I20260926 21:35:42.665191 22602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:42.667080 22613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19cb5293d5c54b2bbf046944edd0d131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb5293d5c54b2bbf046944edd0d131" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33855 } } }
I20260926 21:35:42.667477 22613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:42.670701 22614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19cb5293d5c54b2bbf046944edd0d131. Latest consensus state: current_term: 1 leader_uuid: "19cb5293d5c54b2bbf046944edd0d131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb5293d5c54b2bbf046944edd0d131" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33855 } } }
I20260926 21:35:42.671111 22614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:42.672053 22619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:42.676980 22619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:42.686515 22619 catalog_manager.cc:1347] Generated new cluster ID: 60d0daa564bd4e5b9088769193c15af6
I20260926 21:35:42.686645 22619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:42.695456 22619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:42.697310 22619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:42.714391 22619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19cb5293d5c54b2bbf046944edd0d131: Generated new TSK 0
I20260926 21:35:42.715301 22619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:42.726694 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22535
W20260926 21:35:42.727910 22597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33389
W20260926 21:35:42.728664 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22535: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:42.728816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22535
2026-09-26T21:35:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:42 dist-test-slave-87bt krb5kdc[22633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:42 dist-test-slave-87bt krb5kdc[22633](info): set up 2 sockets
Sep 26 21:35:42 dist-test-slave-87bt krb5kdc[22633](info): commencing operation
krb5kdc: starting...
W20260926 21:35:43.846997 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:44 dist-test-slave-87bt krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:44.106897 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36107
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:44.459605 22646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:44.459918 22646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:44.460098 22646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:44.467746 22646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:44.467890 22646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:44.467978 22646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:44.468087 22646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:44.468168 22646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:44.478358 22646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:44.479704 22646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:44.481114 22646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:44.492952 22654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:44.492918 22652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:44.493638 22651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:44.493643 22646 server_base.cc:1034] running on GCE node
I20260926 21:35:44.494529 22646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:44.495924 22646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:44.497164 22646 hybrid_clock.cc:648] HybridClock initialized: now 1790458544497114 us; error 44 us; skew 500 ppm
I20260926 21:35:44.497537 22646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:44 dist-test-slave-87bt krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:44.500696 22646 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:44.502250 22646 webserver.cc:460] Webserver started at http://127.0.97.62:38171/ using document root <none> and password file <none>
I20260926 21:35:44.503026 22646 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:44.503180 22646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:44.503494 22646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:44.506956 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "287a0975060145498152b1fb20950d08"
format_stamp: "Formatted at 2026-09-26 21:35:44 on dist-test-slave-87bt"
server_key: "b379a7cfc1d1d0f4f109748648e0ed78"
server_key_iv: "38afb2e69351f08f7d1520d32de93a91"
server_key_version: "encryptionkey@0"
I20260926 21:35:44.507777 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "287a0975060145498152b1fb20950d08"
format_stamp: "Formatted at 2026-09-26 21:35:44 on dist-test-slave-87bt"
server_key: "b379a7cfc1d1d0f4f109748648e0ed78"
server_key_iv: "38afb2e69351f08f7d1520d32de93a91"
server_key_version: "encryptionkey@0"
I20260926 21:35:44.514222 22646 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 21:35:44.518229 22661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:44.519824 22646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:44.520040 22646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "287a0975060145498152b1fb20950d08"
format_stamp: "Formatted at 2026-09-26 21:35:44 on dist-test-slave-87bt"
server_key: "b379a7cfc1d1d0f4f109748648e0ed78"
server_key_iv: "38afb2e69351f08f7d1520d32de93a91"
server_key_version: "encryptionkey@0"
I20260926 21:35:44.520259 22646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:44.526260 22646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:44.530754 22646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:44.531075 22646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:44.549924 22646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36107
I20260926 21:35:44.549960 22712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36107 every 8 connection(s)
I20260926 21:35:44.551441 22646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:44.552356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22646
I20260926 21:35:44.552534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:44.553077 388 external_mini_cluster.cc:1428] Setting key 99538de5ebfbfadedb235eac62cac752
I20260926 21:35:44.556170 22713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:44.561090 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:44.555046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38395 (local address 127.0.97.62:36107)
0926 21:35:44.555869 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:44.555882 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:44.555927 (+ 45us) server_negotiation.cc:407] Connection header received
0926 21:35:44.557615 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:44.557638 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:44.557987 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:44.558327 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:44.559448 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:44.559455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:44.559469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:44.559487 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:44.559584 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:44.559604 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:44.559804 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:44.559904 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 21:35:44.560138 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":383,"threads_started":1}
I20260926 21:35:44.564952 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:44.569491 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08: Bootstrap starting.
I20260926 21:35:44.572854 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:44.570142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48849 (local address 127.0.97.62:36107)
0926 21:35:44.570338 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:44.570357 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:44.570994 (+ 637us) server_negotiation.cc:407] Connection header received
0926 21:35:44.571141 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:44.571147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:44.571232 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:44.571372 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:44.572225 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:44.572233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:44.572238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:35:44.572256 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:44.572299 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:35:44.572321 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:44.572441 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:44.572511 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 21:35:44.572567 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:35:44.573397 22713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:44.574544 22713 log.cc:824] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:44.577184 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08: No bootstrap required, opened a new log
I20260926 21:35:44.584288 22713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287a0975060145498152b1fb20950d08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } }
I20260926 21:35:44.584697 22713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:35:44.584801 22713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287a0975060145498152b1fb20950d08, State: Initialized, Role: FOLLOWER
I20260926 21:35:44.585407 22713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287a0975060145498152b1fb20950d08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } }
I20260926 21:35:44.585646 22713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:35:44.585795 22713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:35:44.585969 22713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:44.588266 22713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287a0975060145498152b1fb20950d08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } }
I20260926 21:35:44.588770 22713 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 287a0975060145498152b1fb20950d08; no voters:
I20260926 21:35:44.589459 22713 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:35:44.589548 22723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:44.590324 22723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [term 1 LEADER]: Becoming Leader. State: Replica: 287a0975060145498152b1fb20950d08, State: Running, Role: LEADER
I20260926 21:35:44.590934 22723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287a0975060145498152b1fb20950d08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } }
I20260926 21:35:44.591701 22713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:44.593508 22724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "287a0975060145498152b1fb20950d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287a0975060145498152b1fb20950d08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } } }
I20260926 21:35:44.593958 22724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:44.594403 22725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287a0975060145498152b1fb20950d08. Latest consensus state: current_term: 1 leader_uuid: "287a0975060145498152b1fb20950d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287a0975060145498152b1fb20950d08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36107 } } }
I20260926 21:35:44.594798 22725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:44.597247 22728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:44.602095 22728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:35:44.610805 22728 catalog_manager.cc:1347] Generated new cluster ID: 11e4ab45eb2045728c53da65041b339f
I20260926 21:35:44.610939 22728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:44.627660 22728 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:44.628185 22728 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:44.636267 22728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 287a0975060145498152b1fb20950d08: Generated new TSK 0
I20260926 21:35:44.637348 22728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:44.649942 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22646
W20260926 21:35:44.651057 22708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38395
W20260926 21:35:44.651763 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:35:44.651939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22646
2026-09-26T21:35:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1926 ms)
[----------] 18 tests from AuthTokenIssuingTest (35131 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:44 dist-test-slave-87bt krb5kdc[22744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:44 dist-test-slave-87bt krb5kdc[22744](info): set up 2 sockets
Sep 26 21:35:44 dist-test-slave-87bt krb5kdc[22744](info): commencing operation
krb5kdc: starting...
W20260926 21:35:45.771160 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:45 dist-test-slave-87bt krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:46.039047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34173
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34173,127.0.97.61:39617,127.0.97.60:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:46.371625 22757 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:35:46.372025 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:46.372150 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:46.372287 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:46.379750 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:46.379885 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:46.380016 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:46.380103 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:46.380215 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:46.390283 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34173,127.0.97.61:39617,127.0.97.60:34703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:46.391633 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:46.393049 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:46.405000 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:46.404914 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:46.404949 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:46.406052 22757 server_base.cc:1034] running on GCE node
I20260926 21:35:46.406895 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:46.408288 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:46.409530 22757 hybrid_clock.cc:648] HybridClock initialized: now 1790458546409468 us; error 42 us; skew 500 ppm
I20260926 21:35:46.409873 22757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:46 dist-test-slave-87bt krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:46.413125 22757 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:46.414685 22757 webserver.cc:460] Webserver started at http://127.0.97.62:36373/ using document root <none> and password file <none>
I20260926 21:35:46.415616 22757 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:46.415745 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:46.416087 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:46.419154 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "5091335502d347bf93638f5eef541ccb"
format_stamp: "Formatted at 2026-09-26 21:35:46 on dist-test-slave-87bt"
server_key: "157d51a42e33eee16a2fedd5f21548af"
server_key_iv: "a9e5189e7791c67daa952716c2993168"
server_key_version: "encryptionkey@0"
I20260926 21:35:46.419967 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "5091335502d347bf93638f5eef541ccb"
format_stamp: "Formatted at 2026-09-26 21:35:46 on dist-test-slave-87bt"
server_key: "157d51a42e33eee16a2fedd5f21548af"
server_key_iv: "a9e5189e7791c67daa952716c2993168"
server_key_version: "encryptionkey@0"
I20260926 21:35:46.425208 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:35:46.428860 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:46.430195 22757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:35:46.430404 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "5091335502d347bf93638f5eef541ccb"
format_stamp: "Formatted at 2026-09-26 21:35:46 on dist-test-slave-87bt"
server_key: "157d51a42e33eee16a2fedd5f21548af"
server_key_iv: "a9e5189e7791c67daa952716c2993168"
server_key_version: "encryptionkey@0"
I20260926 21:35:46.430701 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:46.447237 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:46.451869 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:46.452101 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:46.469583 22757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34173
I20260926 21:35:46.469605 22823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34173 every 8 connection(s)
I20260926 21:35:46.470988 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:46.474664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22757
I20260926 21:35:46.474818 22824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:46.474871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:46.475485 388 external_mini_cluster.cc:1428] Setting key 3f577b8e0419c4cb4005c7ffd83f6285
I20260926 21:35:46.481259 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:46 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 21:35:46.506102 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.484807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.0.97.62:34173)
0926 21:35:46.485671 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:46.485687 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:46.487183 (+ 1496us) server_negotiation.cc:407] Connection header received
0926 21:35:46.489204 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:46.489214 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:46.489539 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:46.489752 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:46.490946 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:46.492945 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:46.494310 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:46.495350 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:46.495554 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:46.498948 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:46.498960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:46.498975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:46.499016 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:46.501888 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:46.502771 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:46.502777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:46.502780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:46.502924 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:46.503390 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:46.503393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:46.503395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:46.503871 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:46.504022 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:46.504721 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 21:35:46.505064 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":614,"thread_start_us":389,"threads_started":1}
I20260926 21:35:46.506135 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.484807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34173 (local address 127.0.0.1:56748)
0926 21:35:46.486995 (+ 2188us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.487045 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:46.488832 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:46.489778 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:46.489783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:46.490352 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:46.490740 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:46.490757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:46.493097 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:46.493101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:46.494157 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:46.494167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:46.495569 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:46.495574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:46.495691 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:46.496351 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:46.496372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:46.498714 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:46.502010 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:46.502020 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:46.502029 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:46.502630 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:46.503042 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:46.503050 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:46.503056 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:46.503265 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:46.504016 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:46.504027 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:46.504314 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 21:35:46.504609 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 21:35:46.505248 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1932,"mutex_wait_us":111,"spinlock_wait_cycles":15232,"thread_start_us":1711,"threads_started":1}
I20260926 21:35:46.509506 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:35:46.510763 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.510261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49892)
0926 21:35:46.510499 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.510633 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 21:35:46.511024 22776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
I20260926 21:35:46.512529 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.512213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49898)
0926 21:35:46.512354 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.512424 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 21:35:46.512770 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39617: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 21:35:46.533048 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39617
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34173,127.0.97.61:39617,127.0.97.60:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:35:46.562141 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } attempt: 1
I20260926 21:35:46.563376 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.562920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49904)
0926 21:35:46.563125 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.563240 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 21:35:46.564392 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.564086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49914)
0926 21:35:46.564211 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.564278 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:35:46.564661 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39617: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
I20260926 21:35:46.632913 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } attempt: 2
I20260926 21:35:46.634102 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.633640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49930)
0926 21:35:46.633879 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.633972 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 21:35:46.635169 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.634833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49936)
0926 21:35:46.634973 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.635038 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 21:35:46.635355 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39617: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
I20260926 21:35:46.726675 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } attempt: 3
I20260926 21:35:46.727897 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.727454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49950)
0926 21:35:46.727662 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.727759 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 21:35:46.729236 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.728817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49954)
0926 21:35:46.728981 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.729094 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 21:35:46.729557 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39617: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
W20260926 21:35:46.869292 22831 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:35:46.869661 22831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:46.869788 22831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:46.869930 22831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 21:35:46.872881 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } attempt: 4
I20260926 21:35:46.874127 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.873592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49964)
0926 21:35:46.873910 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.874000 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260926 21:35:46.875142 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:46.874801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49976)
0926 21:35:46.874954 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:46.875030 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 21:35:46.875363 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39617: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39617: connect: Connection refused (error 111)
W20260926 21:35:46.877923 22831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:46.878050 22831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:46.878170 22831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:46.878263 22831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:46.878350 22831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:46.888489 22831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34173,127.0.97.61:39617,127.0.97.60:34703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:46.889884 22831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:46.891309 22831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:46.903264 22839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:46.903229 22836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:46.903316 22837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:46.908054 22831 server_base.cc:1034] running on GCE node
I20260926 21:35:46.908816 22831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:46.910162 22831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:46.911382 22831 hybrid_clock.cc:648] HybridClock initialized: now 1790458546911289 us; error 66 us; skew 500 ppm
I20260926 21:35:46.911713 22831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:46 dist-test-slave-87bt krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:46.914831 22831 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 21:35:46.916270 22831 webserver.cc:460] Webserver started at http://127.0.97.61:44545/ using document root <none> and password file <none>
I20260926 21:35:46.916954 22831 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:46.917042 22831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:46.917317 22831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:46.920395 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/data/instance:
uuid: "68956495520246e68af0ec6a5d0fe903"
format_stamp: "Formatted at 2026-09-26 21:35:46 on dist-test-slave-87bt"
server_key: "c9ebf5a258b90e321966be55fdcb8a03"
server_key_iv: "371a9b16798fce4964a382146bbaa5dd"
server_key_version: "encryptionkey@0"
I20260926 21:35:46.921126 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/wal/instance:
uuid: "68956495520246e68af0ec6a5d0fe903"
format_stamp: "Formatted at 2026-09-26 21:35:46 on dist-test-slave-87bt"
server_key: "c9ebf5a258b90e321966be55fdcb8a03"
server_key_iv: "371a9b16798fce4964a382146bbaa5dd"
server_key_version: "encryptionkey@0"
I20260926 21:35:46.926254 22831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:35:46.929927 22846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:46.931533 22831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:35:46.931732 22831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/wal
uuid: "68956495520246e68af0ec6a5d0fe903"
format_stamp: "Formatted at 2026-09-26 21:35:46 on dist-test-slave-87bt"
server_key: "c9ebf5a258b90e321966be55fdcb8a03"
server_key_iv: "371a9b16798fce4964a382146bbaa5dd"
server_key_version: "encryptionkey@0"
I20260926 21:35:46.931967 22831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:46.963137 22831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:46.967628 22831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:46.967947 22831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:46.987106 22831 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39617
I20260926 21:35:46.987142 22897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39617 every 8 connection(s)
I20260926 21:35:46.988507 22831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:35:46.989419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22831
I20260926 21:35:46.989598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-1/wal/instance
I20260926 21:35:46.990154 388 external_mini_cluster.cc:1428] Setting key e3c1df8872932418334c947fd7e1a029
I20260926 21:35:46.993373 22898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:47.002823 22898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:47.028559 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.005688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.0.97.62:34173)
0926 21:35:47.005886 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.005892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.007367 (+ 1475us) server_negotiation.cc:407] Connection header received
0926 21:35:47.009342 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.009348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.009442 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.009552 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.011253 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.011928 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.013614 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.014035 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.014169 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.021049 (+ 6880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.021059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.021066 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.021123 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.024910 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.025816 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.025826 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.025832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.026000 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.026466 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.026475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.026481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.026914 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.027077 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.027855 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.028126 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:35:47.029281 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.006313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34173 (local address 127.0.0.1:56762)
0926 21:35:47.007184 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.007224 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:47.009046 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:47.009841 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:47.009850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:47.010409 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:47.011070 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.011087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.012098 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.012102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.013454 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.013471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.014278 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.014285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.014591 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.015360 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:47.015398 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:47.020724 (+ 5326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:47.025110 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.025119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.025135 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.025646 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.026113 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.026116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.026118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.026311 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.027062 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:47.027070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:47.027440 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 21:35:47.027676 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 21:35:47.028124 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":371,"threads_started":1}
I20260926 21:35:47.031217 22898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:35:47.049005 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.032089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:49990)
0926 21:35:47.032317 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.032338 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:47.032454 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:47.033368 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:47.033379 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:47.033894 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:47.034103 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.034117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.035741 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.035750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.037349 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.037365 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.038190 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.038198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.038459 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.039087 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:47.039114 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:47.041679 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:47.046374 (+ 4695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.046379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.046381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.046901 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.047369 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.047373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.047375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.047504 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.048154 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:47.048158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:47.048301 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 21:35:47.048416 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 21:35:47.048731 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 21:35:47.049278 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.032293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.0.97.61:39617)
0926 21:35:47.032952 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.032958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.032982 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:35:47.033025 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.033031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.033205 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.033326 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.034281 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.035593 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.037486 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.038025 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.038125 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.041942 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.041963 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.041984 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.042032 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.046171 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.047058 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.047070 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.047076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.047227 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.047608 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.047612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.047613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.048016 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.048198 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.048564 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.048872 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":499,"thread_start_us":376,"threads_started":1}
I20260926 21:35:47.051656 22898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:35:47.052760 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.052303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60320)
0926 21:35:47.052487 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.052599 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 21:35:47.053045 22850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
I20260926 21:35:47.054618 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.054195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60336)
0926 21:35:47.054383 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.054517 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 21:35:47.054844 22898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 21:35:47.057847 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34703
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34173,127.0.97.61:39617,127.0.97.60:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:35:47.104116 22898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 1
I20260926 21:35:47.105311 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.104751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60352)
0926 21:35:47.104988 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.105123 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 21:35:47.106379 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.105990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60354)
0926 21:35:47.106147 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.106247 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 21:35:47.106590 22898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
I20260926 21:35:47.174623 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } attempt: 5
I20260926 21:35:47.174847 22898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 2
I20260926 21:35:47.176105 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.175464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60370)
0926 21:35:47.175794 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.175895 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260926 21:35:47.177309 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.176871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60382)
0926 21:35:47.177036 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.177132 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 21:35:47.178532 22898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:35:47.189030 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.175459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:50002)
0926 21:35:47.175667 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.175688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:47.175831 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:47.176205 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:47.176211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:47.176759 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:47.176984 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.176997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.178406 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.178411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.179435 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.179447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.180390 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.180394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.180530 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.181169 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:47.181187 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:47.183255 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:47.186835 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.186839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.186842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.187193 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.187596 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.187599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.187602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.187724 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.188322 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:47.188325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:47.188459 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 21:35:47.188576 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 21:35:47.188790 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 21:35:47.189244 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.175694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.0.97.61:39617)
0926 21:35:47.175827 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.175832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.175852 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:35:47.175964 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.175969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.176059 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.176170 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.177147 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.178247 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.179579 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.180242 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.180372 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.183459 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.183466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.183470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.183509 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.186704 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.187312 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.187317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.187321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.187471 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.187829 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.187834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.187837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.188207 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.188355 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.188714 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.188939 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 21:35:47.190246 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:35:47.191326 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.190961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60386)
0926 21:35:47.191123 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.191195 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 21:35:47.192257 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.191953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60396)
0926 21:35:47.192093 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.192167 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 21:35:47.192456 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
I20260926 21:35:47.243718 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 1
I20260926 21:35:47.244987 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.244515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60404)
0926 21:35:47.244723 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.244820 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 21:35:47.245991 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.245658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60408)
0926 21:35:47.245814 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.245891 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 21:35:47.246194 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
I20260926 21:35:47.269759 22898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 3
I20260926 21:35:47.271080 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.270539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60420)
0926 21:35:47.270777 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.270867 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 21:35:47.272161 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.271743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60436)
0926 21:35:47.271964 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.272048 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260926 21:35:47.272353 22898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
I20260926 21:35:47.314469 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 2
I20260926 21:35:47.315544 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.315082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60442)
0926 21:35:47.315308 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.315388 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 21:35:47.316561 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.316241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60446)
0926 21:35:47.316399 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.316467 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 21:35:47.316758 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
W20260926 21:35:47.401880 22905 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:35:47.402252 22905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:47.402379 22905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:47.402531 22905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:47.409763 22905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:47.409897 22905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:47.410025 22905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:47.410110 22905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:47.410207 22905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:47.415548 22898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 4
I20260926 21:35:47.416764 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.416319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60458)
0926 21:35:47.416535 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.416618 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 21:35:47.417711 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.417389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60466)
0926 21:35:47.417535 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.417611 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:35:47.417912 22898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
I20260926 21:35:47.420907 22905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34173,127.0.97.61:39617,127.0.97.60:34703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:47.422252 22905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:47.422951 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 3
I20260926 21:35:47.423782 22905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 21:35:47.423954 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.423486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60474)
0926 21:35:47.423703 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.423784 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 21:35:47.424948 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.424596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60488)
0926 21:35:47.424787 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.424854 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 21:35:47.425128 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34703: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34703: connect: Connection refused (error 111)
W20260926 21:35:47.436285 22910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:47.436398 22911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:47.436307 22913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:47.437486 22905 server_base.cc:1034] running on GCE node
I20260926 21:35:47.438263 22905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:47.439687 22905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:47.440932 22905 hybrid_clock.cc:648] HybridClock initialized: now 1790458547440880 us; error 31 us; skew 500 ppm
I20260926 21:35:47.441282 22905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:47.444429 22905 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 21:35:47.445839 22905 webserver.cc:460] Webserver started at http://127.0.97.60:34465/ using document root <none> and password file <none>
I20260926 21:35:47.446647 22905 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:47.446779 22905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:47.447110 22905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:47.450007 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/data/instance:
uuid: "8516bb85de4c4fdcbc3d210a9a9c8820"
format_stamp: "Formatted at 2026-09-26 21:35:47 on dist-test-slave-87bt"
server_key: "281329193d90632cba2856124d009409"
server_key_iv: "f8107a3a504a993fe000cc04bc265e3b"
server_key_version: "encryptionkey@0"
I20260926 21:35:47.450850 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/wal/instance:
uuid: "8516bb85de4c4fdcbc3d210a9a9c8820"
format_stamp: "Formatted at 2026-09-26 21:35:47 on dist-test-slave-87bt"
server_key: "281329193d90632cba2856124d009409"
server_key_iv: "f8107a3a504a993fe000cc04bc265e3b"
server_key_version: "encryptionkey@0"
I20260926 21:35:47.455861 22905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:35:47.459213 22920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:47.460525 22905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:35:47.460696 22905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/wal
uuid: "8516bb85de4c4fdcbc3d210a9a9c8820"
format_stamp: "Formatted at 2026-09-26 21:35:47 on dist-test-slave-87bt"
server_key: "281329193d90632cba2856124d009409"
server_key_iv: "f8107a3a504a993fe000cc04bc265e3b"
server_key_version: "encryptionkey@0"
I20260926 21:35:47.460908 22905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:47.473850 22905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:47.478891 22905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:47.479149 22905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:47.496235 22905 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34703
I20260926 21:35:47.496255 22971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34703 every 8 connection(s)
I20260926 21:35:47.497587 22905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:35:47.501236 22972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:47.502868 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 22905
I20260926 21:35:47.503032 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458187148965-388-0/minicluster-data/master-2/wal/instance
I20260926 21:35:47.503466 388 external_mini_cluster.cc:1428] Setting key 0239033317ba490690027c38672abe23
I20260926 21:35:47.510789 22972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:47.527913 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.505301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.0.97.60:34703)
0926 21:35:47.506107 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.506122 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.506169 (+ 47us) server_negotiation.cc:407] Connection header received
0926 21:35:47.507858 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.507886 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.508393 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.508832 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.510035 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.512078 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.513498 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.514398 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.514653 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.519177 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.519186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.519199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.519241 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.523103 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.523935 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.523950 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.523958 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.524116 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.524538 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.524543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.524548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.525114 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.525431 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.525915 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.526410 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":352,"threads_started":1}
I20260926 21:35:47.529327 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.513694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.0.97.62:34173)
0926 21:35:47.513937 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.513943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.514696 (+ 753us) server_negotiation.cc:407] Connection header received
0926 21:35:47.514944 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.514950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.515020 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.515129 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.516211 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.516900 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.518692 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.519124 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.519285 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.523302 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.523307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.523311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.523351 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.526224 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.527125 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.527133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.527140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.527290 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.527712 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.527716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.527718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.528009 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.528167 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.528735 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.528955 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:35:47.529385 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.513662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34173 (local address 127.0.0.1:56774)
0926 21:35:47.514568 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.514616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:47.514818 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:47.515191 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:47.515199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:47.515830 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:47.516063 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.516071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.517179 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.517185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.518494 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.518508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.519342 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.519349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.519551 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.520305 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:47.520323 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:47.523036 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:47.526372 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.526383 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.526390 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.526970 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.527411 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.527416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.527420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.527567 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.528223 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:47.528232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:47.528418 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 21:35:47.528602 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 21:35:47.528934 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":486,"threads_started":1}
I20260926 21:35:47.531191 22972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:47.545883 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.532044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39617 (local address 127.0.0.1:50012)
0926 21:35:47.532260 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.532280 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:47.532382 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:47.532694 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:47.532700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:47.533175 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:47.533375 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.533386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.534738 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.534744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.536226 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.536240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.537018 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.537024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.537225 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.537850 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:47.537868 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:47.540119 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:47.543479 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.543485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.543489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.543858 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.544249 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.544255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.544259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.544411 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.544928 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:47.544933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:47.545090 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 21:35:47.545228 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 21:35:47.545525 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 21:35:47.545915 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.532141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.0.97.61:39617)
0926 21:35:47.532320 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.532324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.532339 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:35:47.532497 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.532501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.532556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.532638 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.533531 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.534584 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.536407 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.536858 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.536995 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.540316 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.540325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.540334 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.540366 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.543342 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.543991 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.544002 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.544005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.544132 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.544520 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.544527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.544530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.544808 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.544963 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.545327 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.545533 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:35:47.547010 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.531333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.0.97.62:34173)
0926 21:35:47.531498 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.531504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.531522 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:35:47.531624 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.531630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.531698 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.531801 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.533350 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.534560 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.536458 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.536868 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.537011 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.541325 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.541330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.541333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.541372 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.544139 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.544810 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.544817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.544824 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.544942 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.545306 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.545311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.545315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.545580 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.545868 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.546583 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.546772 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:35:47.547127 22972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:35:47.562214 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.547811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60498)
0926 21:35:47.548112 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.548129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:47.548234 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:47.548527 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:47.548531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:47.549003 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:47.549174 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.549188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.550660 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.550665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.552016 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.552027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.552933 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.552938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.553089 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.553650 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:47.553673 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:47.556129 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:47.559478 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.559483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.559486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.560057 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.560438 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.560441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.560443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.560624 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.561286 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:47.561289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:47.561464 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 21:35:47.561670 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 21:35:47.561908 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260926 21:35:47.562525 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.547954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.0.97.60:34703)
0926 21:35:47.548113 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.548120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.548172 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:35:47.548352 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.548357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.548421 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.548513 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.549328 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.550512 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.552140 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.552765 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.552894 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.556332 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.556340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.556347 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.556386 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.559331 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.560180 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.560186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.560190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.560327 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.560767 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.560772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.560775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.561158 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.561324 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.561785 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.562082 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":52}
I20260926 21:35:47.565192 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.548655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.0.97.61:39617)
0926 21:35:47.548842 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.548848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.548886 (+ 38us) server_negotiation.cc:407] Connection header received
0926 21:35:47.548981 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.548986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.549050 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.549141 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.550664 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.551760 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.553316 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.553911 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.554044 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.558978 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.558984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.558989 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.559036 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.562482 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.563233 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.563238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.563242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.563395 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.563802 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.563810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.563813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.564087 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.564380 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.564642 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.564863 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:35:47.568540 22972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: Bootstrap starting.
I20260926 21:35:47.570056 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:47.572666 22972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:47.573863 22972 log.cc:824] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:47.572779 22799 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5091335502d347bf93638f5eef541ccb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56786
I20260926 21:35:47.576951 22972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: No bootstrap required, opened a new log
I20260926 21:35:47.584478 22972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } }
I20260926 21:35:47.584882 22972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8516bb85de4c4fdcbc3d210a9a9c8820, State: Initialized, Role: FOLLOWER
I20260926 21:35:47.585363 22972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } }
I20260926 21:35:47.586248 22978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.586624 22978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:47.587782 22972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:35:47.597314 22989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:35:47.597436 22989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:35:47.602311 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 4
I20260926 21:35:47.595186 22905 master_runner.cc:186] Error getting master registration for 127.0.97.62:34173: OK, instance_id { permanent_uuid: "5091335502d347bf93638f5eef541ccb" instance_seqno: 1790458546464625 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:35:47.607154 22905 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:35:47.619058 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.603004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60512)
0926 21:35:47.603200 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.603223 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:47.603352 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:47.603692 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:47.603697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:47.604215 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:47.604426 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.604437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.605928 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.605934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.607354 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.607366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.608344 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.608349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.608552 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.609236 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:47.609262 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:47.612050 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:47.616199 (+ 4149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.616208 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.616211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.616592 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.617050 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.617055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.617059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.617271 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.617984 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:47.617988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:47.618163 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 21:35:47.618312 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 21:35:47.618648 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 21:35:47.619426 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.603065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.0.97.60:34703)
0926 21:35:47.603361 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.603367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.603389 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:35:47.603451 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.603456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.603534 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.603644 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.604581 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.605764 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.607491 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.608176 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.608306 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.612265 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.612272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.612277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.612317 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.616051 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.616716 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.616722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.616726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.616922 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.617388 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.617393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.617397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.617857 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.618024 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.618489 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.618732 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:35:47.625412 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb: Bootstrap starting.
I20260926 21:35:47.628702 22824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:47.629791 22824 log.cc:824] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:47.633525 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb: No bootstrap required, opened a new log
I20260926 21:35:47.639199 22824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } }
I20260926 21:35:47.639557 22824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5091335502d347bf93638f5eef541ccb, State: Initialized, Role: FOLLOWER
I20260926 21:35:47.640024 22824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } }
I20260926 21:35:47.640969 22992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.641321 22992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:47.642103 22824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:47.642267 22799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 21:35:47.642549 22799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:47.646486 22799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } }
I20260926 21:35:47.650940 22947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5091335502d347bf93638f5eef541ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820"
I20260926 21:35:47.651474 22947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:47.654930 22947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5091335502d347bf93638f5eef541ccb in term 1.
I20260926 21:35:47.655681 22776 leader_election.cc:302] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5091335502d347bf93638f5eef541ccb, 8516bb85de4c4fdcbc3d210a9a9c8820; no voters:
I20260926 21:35:47.656188 22992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:47.656720 22799 leader_election.cc:288] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [CANDIDATE]: Term 1 election: Requested vote from peers 68956495520246e68af0ec6a5d0fe903 (127.0.97.61:39617), 8516bb85de4c4fdcbc3d210a9a9c8820 (127.0.97.60:34703)
I20260926 21:35:47.668342 22992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [term 1 LEADER]: Becoming Leader. State: Replica: 5091335502d347bf93638f5eef541ccb, State: Running, Role: LEADER
I20260926 21:35:47.669200 22992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } }
W20260926 21:35:47.676836 23003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:35:47.676972 23003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:35:47.683367 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5091335502d347bf93638f5eef541ccb. Latest consensus state: current_term: 1 leader_uuid: "5091335502d347bf93638f5eef541ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.683921 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:47.652726 22757 master_runner.cc:186] Error getting master registration for 127.0.97.61:39617: OK, instance_id { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" instance_seqno: 1790458546981854 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:35:47.687613 22757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:35:47.694137 23012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:47.700493 23012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 21:35:47.704944 22778 tablet.cc:2369] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:35:47.712355 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.681975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.0.97.62:34173)
0926 21:35:47.682164 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.682169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.688071 (+ 5902us) server_negotiation.cc:407] Connection header received
0926 21:35:47.688277 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.688283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.688362 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.688460 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.692499 (+ 4039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.693635 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.696775 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.697504 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.697609 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.703275 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.703282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.703287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.703330 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.707428 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.708771 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.708778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.708782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.708959 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.709628 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.709634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.709637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.710037 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.710285 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.711297 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.711705 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:35:47.717094 22898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } attempt: 5
I20260926 21:35:47.724375 22947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [term 1 FOLLOWER]: Refusing update from remote peer 5091335502d347bf93638f5eef541ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 21:35:47.731189 22926 tablet.cc:2369] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:35:47.732651 23005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:47.735137 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.688538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.97.61:39617)
0926 21:35:47.688708 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.688714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.694792 (+ 6078us) server_negotiation.cc:407] Connection header received
0926 21:35:47.696128 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.696135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.696220 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.696290 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.700712 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.701878 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.710720 (+ 8842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.711420 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.711520 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.716189 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.716196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.716203 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.716249 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.721431 (+ 5182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.729360 (+ 7929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.729369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.729372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.729567 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.731155 (+ 1588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.731162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.731165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.731739 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.732010 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.734367 (+ 2357us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.734750 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:35:47.752792 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.695331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.0.97.60:34703)
0926 21:35:47.695505 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.695513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.713536 (+ 18023us) server_negotiation.cc:407] Connection header received
0926 21:35:47.716317 (+ 2781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.716324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.716407 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.716475 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.718342 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.719403 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.721264 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.722005 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.722102 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.728202 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.728210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.728215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.728266 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.732127 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.744323 (+ 12196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.744339 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.744344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.744885 (+ 541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.748561 (+ 3676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.748570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.748576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.749094 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.749445 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.751247 (+ 1802us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.751525 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:35:47.758821 22978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5091335502d347bf93638f5eef541ccb. Latest consensus state: current_term: 1 leader_uuid: "5091335502d347bf93638f5eef541ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.759202 22978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 21:35:47 dist-test-slave-87bt krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:35:47.772459 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.718512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34703 (local address 127.0.0.1:60528)
0926 21:35:47.718908 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:47.718926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:47.719022 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:47.746483 (+ 27461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:47.746489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:47.747055 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:47.747282 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.747296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.756237 (+ 8941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.756248 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.757954 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:47.757974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.760233 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.760240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:47.760466 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.761471 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:47.761496 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:47.764492 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:47.769415 (+ 4923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.769421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.769425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.769951 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.770518 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:47.770523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:47.770527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:47.770695 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:47.771394 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:47.771402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:47.771565 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 21:35:47.771701 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 21:35:47.772032 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260926 21:35:47.773478 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:47.719353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.0.97.60:34703)
0926 21:35:47.745848 (+ 26495us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:47.745854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:47.745910 (+ 56us) server_negotiation.cc:407] Connection header received
0926 21:35:47.746001 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:47.746041 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:47.746197 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:47.746415 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:47.747853 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.755893 (+ 8040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.758262 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:47.759923 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:47.760856 (+ 933us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:47.764814 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:47.764821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:47.764827 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:47.764864 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:47.769251 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.770120 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.770130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.770134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.770343 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:47.770845 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:47.770851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:47.770854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:47.771239 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:47.772695 (+ 1456us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:47.772849 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 21:35:47.773078 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26212,"thread_start_us":25090,"threads_started":1}
I20260926 21:35:47.776382 23017 mvcc.cc:205] Tried to move back new op lower bound from 7333718211460141056 to 7333718211262324736. Current Snapshot: MvccSnapshot[applied={T|T < 7333718211460141056}]
I20260926 21:35:47.784137 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5091335502d347bf93638f5eef541ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.784451 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:47.788103 23012 catalog_manager.cc:1347] Generated new cluster ID: 10210d175b4c48c9bbd021ae57cbc88b
I20260926 21:35:47.788241 23012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:47.798031 22898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903: Bootstrap starting.
I20260926 21:35:47.815692 22978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5091335502d347bf93638f5eef541ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.815972 22978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:47.831904 22898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:47.833540 22898 log.cc:824] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:47.846081 22898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903: No bootstrap required, opened a new log
I20260926 21:35:47.858742 22898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } }
I20260926 21:35:47.860471 23012 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:47.860970 22898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68956495520246e68af0ec6a5d0fe903, State: Initialized, Role: FOLLOWER
I20260926 21:35:47.861614 22898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } }
I20260926 21:35:47.862342 23012 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:47.868942 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.871352 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:47.871227 22898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:47.871868 22872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:47.875768 22872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [term 1 FOLLOWER]: Refusing update from remote peer 5091335502d347bf93638f5eef541ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:47.876729 22992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:47.877846 22873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5091335502d347bf93638f5eef541ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "68956495520246e68af0ec6a5d0fe903"
I20260926 21:35:47.881412 22873 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5091335502d347bf93638f5eef541ccb in term 1.
I20260926 21:35:47.894893 23012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb: Generated new TSK 0
I20260926 21:35:47.896050 23012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:47.949589 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5091335502d347bf93638f5eef541ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.950085 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:47.949909 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5091335502d347bf93638f5eef541ccb. Latest consensus state: current_term: 1 leader_uuid: "5091335502d347bf93638f5eef541ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.950214 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:47.955065 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5091335502d347bf93638f5eef541ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5091335502d347bf93638f5eef541ccb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34173 } } peers { permanent_uuid: "68956495520246e68af0ec6a5d0fe903" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39617 } } peers { permanent_uuid: "8516bb85de4c4fdcbc3d210a9a9c8820" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34703 } } }
I20260926 21:35:47.955382 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5091335502d347bf93638f5eef541ccb [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:47.973482 23039 catalog_manager.cc:1259] Loaded cluster ID: 10210d175b4c48c9bbd021ae57cbc88b
I20260926 21:35:47.973613 23039 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903: loading cluster ID for follower catalog manager: success
I20260926 21:35:47.978003 23039 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903: acquiring CA information for follower catalog manager: success
I20260926 21:35:47.980736 23039 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68956495520246e68af0ec6a5d0fe903: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 21:35:48.038234 22827 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.019930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.0.97.62:34173)
0926 21:35:48.020132 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.020139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.036125 (+ 15986us) server_negotiation.cc:407] Connection header received
0926 21:35:48.036551 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.036559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.036665 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.036744 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.037964 (+ 1220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56792: BlockingRecv error: recv got EOF from 127.0.0.1:56792 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 21:35:48.040398 22903 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.028312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.0.97.61:39617)
0926 21:35:48.028519 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.028526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.035643 (+ 7117us) server_negotiation.cc:407] Connection header received
0926 21:35:48.038861 (+ 3218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.038867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.038956 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.039032 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.040140 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50034: BlockingRecv error: recv got EOF from 127.0.0.1:50034 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 21:35:48.042650 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.034912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.0.97.60:34703)
0926 21:35:48.035107 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.035114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.041195 (+ 6081us) server_negotiation.cc:407] Connection header received
0926 21:35:48.041361 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.041373 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.041480 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.041618 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.042403 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60542: BlockingRecv error: recv got EOF from 127.0.0.1:60542 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:35:48.092530 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.077073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.0.97.61:39617)
0926 21:35:48.077275 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.077282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.084178 (+ 6896us) server_negotiation.cc:407] Connection header received
0926 21:35:48.084759 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.084766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.085004 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.085170 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.085925 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.087139 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.089266 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.089955 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.090054 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.090912 (+ 858us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.091649 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.091785 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.092053 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.092168 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 21:35:48.136638 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.126763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.0.97.60:34703)
0926 21:35:48.126970 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.126976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.133974 (+ 6998us) server_negotiation.cc:407] Connection header received
0926 21:35:48.134771 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.134779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.134918 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.135003 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.136358 (+ 1355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60546: BlockingRecv error: recv got EOF from 127.0.0.1:60546 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:35:48.185372 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.171175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.0.97.61:39617)
0926 21:35:48.171425 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.171446 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.178464 (+ 7018us) server_negotiation.cc:407] Connection header received
0926 21:35:48.178982 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.178993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.179107 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.179177 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.179855 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.181010 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.183100 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.183783 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.183889 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.184494 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.184662 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.184724 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.184958 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.185053 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 21:35:48.230525 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.221131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.97.60:34703)
0926 21:35:48.221379 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.221386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.228336 (+ 6950us) server_negotiation.cc:407] Connection header received
0926 21:35:48.228899 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.228905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.229007 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.229111 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.230216 (+ 1105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60560: BlockingRecv error: recv got EOF from 127.0.0.1:60560 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 21:35:48.248467 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.240552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.0.97.61:39617)
0926 21:35:48.240753 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.240760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.241302 (+ 542us) server_negotiation.cc:407] Connection header received
0926 21:35:48.241618 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.241625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.241723 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.241795 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.242530 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.243723 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.246130 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.246938 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.247087 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.247551 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.247705 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.247810 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.248012 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.248111 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 21:35:48.258366 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.255873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.97.60:34703)
0926 21:35:48.256071 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.256079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.256604 (+ 525us) server_negotiation.cc:407] Connection header received
0926 21:35:48.256907 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.256913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.257013 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.257105 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.258114 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60564: BlockingRecv error: recv got EOF from 127.0.0.1:60564 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:35:48.273555 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.266396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.0.97.61:39617)
0926 21:35:48.266665 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.266672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.267204 (+ 532us) server_negotiation.cc:407] Connection header received
0926 21:35:48.267410 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.267416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.267514 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.267602 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.268073 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.269154 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.271184 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.271906 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.272041 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.272592 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.272727 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.272832 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.273171 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.273265 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
W20260926 21:35:48.283699 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.281087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.0.97.60:34703)
0926 21:35:48.281320 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.281326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.281898 (+ 572us) server_negotiation.cc:407] Connection header received
0926 21:35:48.282205 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.282211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.282328 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.282409 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.283467 (+ 1058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60566: BlockingRecv error: recv got EOF from 127.0.0.1:60566 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 21:35:48.303161 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.296045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.0.97.61:39617)
0926 21:35:48.296240 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.296246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.296781 (+ 535us) server_negotiation.cc:407] Connection header received
0926 21:35:48.296981 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.296990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.297084 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.297198 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.297678 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.298915 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.300937 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.301645 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.301807 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.302277 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.302451 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.302551 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.302727 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.302820 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 21:35:48.312520 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.310121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.97.60:34703)
0926 21:35:48.310334 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.310341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.310846 (+ 505us) server_negotiation.cc:407] Connection header received
0926 21:35:48.311127 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.311133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.311227 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.311297 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.312261 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60574: BlockingRecv error: recv got EOF from 127.0.0.1:60574 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:35:48.340827 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.333104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.0.97.61:39617)
0926 21:35:48.333344 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.333351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.334112 (+ 761us) server_negotiation.cc:407] Connection header received
0926 21:35:48.334337 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.334343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.334467 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.334572 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.335146 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.336500 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.338669 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.339380 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.339521 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.340025 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.340178 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.340276 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.340420 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.340528 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 21:35:48.353287 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.351118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.97.60:34703)
0926 21:35:48.351292 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.351298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.351710 (+ 412us) server_negotiation.cc:407] Connection header received
0926 21:35:48.351956 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.351963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.352049 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.352131 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.353031 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60576: BlockingRecv error: recv got EOF from 127.0.0.1:60576 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:35:48.395875 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.389552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.0.97.61:39617)
0926 21:35:48.389794 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.389801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.390136 (+ 335us) server_negotiation.cc:407] Connection header received
0926 21:35:48.390522 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.390530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.390618 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.390733 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.391187 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.392354 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.393714 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.394533 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.394675 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.395043 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.395194 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.395291 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.395445 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.395531 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 21:35:48.403820 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.401699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.97.60:34703)
0926 21:35:48.401948 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.401955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.402295 (+ 340us) server_negotiation.cc:407] Connection header received
0926 21:35:48.402518 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.402523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.402591 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.402675 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.403577 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60584: BlockingRecv error: recv got EOF from 127.0.0.1:60584 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:35:48.478813 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.472866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.0.97.61:39617)
0926 21:35:48.473061 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.473068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.473600 (+ 532us) server_negotiation.cc:407] Connection header received
0926 21:35:48.473825 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.473831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.473912 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.473992 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.474508 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.475336 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.477115 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.477521 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.477666 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.478071 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.478175 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.478282 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.478455 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.478540 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 21:35:48.487164 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:48.485109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.0.97.60:34703)
0926 21:35:48.485321 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.485329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.485605 (+ 276us) server_negotiation.cc:407] Connection header received
0926 21:35:48.485776 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.485780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.485851 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.485944 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:48.486909 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60588: BlockingRecv error: recv got EOF from 127.0.0.1:60588 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:35:48.599457 22989 catalog_manager.cc:1259] Loaded cluster ID: 10210d175b4c48c9bbd021ae57cbc88b
I20260926 21:35:48.599560 22989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: loading cluster ID for follower catalog manager: success
I20260926 21:35:48.602178 22989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: acquiring CA information for follower catalog manager: success
I20260926 21:35:48.603992 22989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8516bb85de4c4fdcbc3d210a9a9c8820: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:35:48.623804 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.618634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.0.97.61:39617)
0926 21:35:48.618898 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.618906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.619223 (+ 317us) server_negotiation.cc:407] Connection header received
0926 21:35:48.619447 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.619458 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.619536 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.619638 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.620090 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.620787 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.622102 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.622567 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.622696 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.623101 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.623206 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.623315 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.623454 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.623534 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 21:35:48.635138 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.629702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.0.97.60:34703)
0926 21:35:48.629878 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.629883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.630253 (+ 370us) server_negotiation.cc:407] Connection header received
0926 21:35:48.630459 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.630469 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.630630 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.630730 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.631093 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.631790 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.632977 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.633385 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.633469 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.633949 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.634563 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.634673 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.634766 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.634866 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:35:48.650039 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.641884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.97.61:39617)
0926 21:35:48.642335 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.642364 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.642924 (+ 560us) server_negotiation.cc:407] Connection header received
0926 21:35:48.643139 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.643148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.643293 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.643420 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.644175 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.645620 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.647894 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.648567 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.648704 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.649223 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.649362 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.649448 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.649641 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.649725 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260926 21:35:48.652010 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.642546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.0.97.60:34703)
0926 21:35:48.642769 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.642777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.645785 (+ 3008us) server_negotiation.cc:407] Connection header received
0926 21:35:48.646092 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.646098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.646190 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.646301 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.647057 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.648048 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.649927 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.650586 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.650680 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.651261 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.651388 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.651446 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.651659 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.651723 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:35:48.652711 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:48.641330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.0.97.62:34173)
0926 21:35:48.645409 (+ 4079us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:48.645418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:48.645444 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:35:48.645570 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:48.645576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:48.645826 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:48.646012 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:48.646701 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.647604 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.649460 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:48.650113 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:48.650222 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:48.651319 (+ 1097us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:48.651770 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:48.651869 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:48.652360 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 21:35:48.652444 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3941,"thread_start_us":3845,"threads_started":1}
I20260926 21:35:48.657317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22757
I20260926 21:35:48.658293 22819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:48.767987 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22757
I20260926 21:35:48.788362 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22831
I20260926 21:35:48.789464 22893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:48.900699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22831
I20260926 21:35:48.920751 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 22905
I20260926 21:35:48.921854 22967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:49.031176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 22905
2026-09-26T21:35:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4384 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:35:49 dist-test-slave-87bt krb5kdc[23152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:35:49 dist-test-slave-87bt krb5kdc[23152](info): set up 2 sockets
Sep 26 21:35:49 dist-test-slave-87bt krb5kdc[23152](info): commencing operation
krb5kdc: starting...
W20260926 21:35:50.147176 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/test-user.keytab.
Sep 26 21:35:50 dist-test-slave-87bt krb5kdc[23152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:35:50.417774 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35973
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35973,127.0.97.61:40971,127.0.97.60:46693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:35:50.762358 23165 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:35:50.762825 23165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:50.762945 23165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:50.763077 23165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:50.770659 23165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:50.770797 23165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:50.770916 23165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:50.770999 23165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:50.771103 23165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:50.781469 23165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35973,127.0.97.61:40971,127.0.97.60:46693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:50.782950 23165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:50.784296 23165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:50.796573 23170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:50.796521 23171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:50.796597 23173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:50.797501 23165 server_base.cc:1034] running on GCE node
I20260926 21:35:50.798398 23165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:50.799979 23165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:50.801275 23165 hybrid_clock.cc:648] HybridClock initialized: now 1790458550801180 us; error 50 us; skew 500 ppm
I20260926 21:35:50.801642 23165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:50 dist-test-slave-87bt krb5kdc[23152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:50.811075 23165 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:35:50.812637 23165 webserver.cc:460] Webserver started at http://127.0.97.62:35963/ using document root <none> and password file <none>
I20260926 21:35:50.813367 23165 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:50.813458 23165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:50.813737 23165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:50.816982 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/data/instance:
uuid: "0af5008a18fc4aca9139489044346f11"
format_stamp: "Formatted at 2026-09-26 21:35:50 on dist-test-slave-87bt"
server_key: "c0892465b39fa07a4570d2db7919b014"
server_key_iv: "ece6e9e1ac7863cf910b0a4d2f2d220d"
server_key_version: "encryptionkey@0"
I20260926 21:35:50.817848 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/wal/instance:
uuid: "0af5008a18fc4aca9139489044346f11"
format_stamp: "Formatted at 2026-09-26 21:35:50 on dist-test-slave-87bt"
server_key: "c0892465b39fa07a4570d2db7919b014"
server_key_iv: "ece6e9e1ac7863cf910b0a4d2f2d220d"
server_key_version: "encryptionkey@0"
I20260926 21:35:50.823875 23165 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 21:35:50.828159 23180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:50.830031 23165 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 21:35:50.830332 23165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/wal
uuid: "0af5008a18fc4aca9139489044346f11"
format_stamp: "Formatted at 2026-09-26 21:35:50 on dist-test-slave-87bt"
server_key: "c0892465b39fa07a4570d2db7919b014"
server_key_iv: "ece6e9e1ac7863cf910b0a4d2f2d220d"
server_key_version: "encryptionkey@0"
I20260926 21:35:50.830765 23165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:50.849874 23165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:50.854652 23165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:50.854949 23165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:50.874265 23165 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35973
I20260926 21:35:50.874289 23231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35973 every 8 connection(s)
I20260926 21:35:50.875725 23165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:35:50.879535 23232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:50.884279 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 23165
I20260926 21:35:50.884470 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-0/wal/instance
I20260926 21:35:50.885020 388 external_mini_cluster.cc:1428] Setting key eaa30e4f99b58a506f5af8f153339a3e
I20260926 21:35:50.888219 23232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:50 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260926 21:35:50.914112 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:50.892404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35973 (local address 127.0.0.1:33544)
0926 21:35:50.893355 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:50.893401 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:50.895504 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:50.896392 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:50.896400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:50.897273 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:50.897831 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:50.897860 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:50.899641 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:50.899650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:50.901618 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:50.901639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:50.902698 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:50.902707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:50.902931 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:50.903667 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:50.903703 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:50.906510 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:50.909943 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:50.909954 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:50.909967 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:50.910388 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:50.910950 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:50.910953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:50.910955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:50.911100 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:50.912184 (+ 1084us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:50.912191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:50.912384 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 21:35:50.912593 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 21:35:50.913020 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":431,"threads_started":1}
I20260926 21:35:50.914178 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:50.892405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.0.97.62:35973)
0926 21:35:50.894176 (+ 1771us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:50.894185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:50.894239 (+ 54us) server_negotiation.cc:407] Connection header received
0926 21:35:50.895899 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:50.895904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:50.896170 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:50.896384 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:50.898108 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:50.899447 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:50.901796 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:50.902394 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:50.902612 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:50.906755 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:50.906764 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:50.906777 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:50.906808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:50.909793 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:50.910579 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:50.910591 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:50.910597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:50.910824 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:50.911226 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:50.911232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:50.911238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:50.912032 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:50.912260 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:50.912810 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 21:35:50.913230 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":1547,"thread_start_us":1326,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 21:35:50.917263 23232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:35:50.918566 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:50.918090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53722)
0926 21:35:50.918283 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:50.918387 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 21:35:50.918849 23182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
I20260926 21:35:50.920558 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:50.920127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53730)
0926 21:35:50.920316 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:50.920390 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 21:35:50.920801 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 21:35:50.950174 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40971
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35973,127.0.97.61:40971,127.0.97.60:46693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:35:50.970228 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } attempt: 1
I20260926 21:35:50.971666 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:50.971039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53744)
0926 21:35:50.971311 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:50.971452 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 21:35:50.972812 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:50.972414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53754)
0926 21:35:50.972561 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:50.972626 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 21:35:50.973104 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
I20260926 21:35:51.041347 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } attempt: 2
I20260926 21:35:51.042547 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.042055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53756)
0926 21:35:51.042244 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.042334 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 21:35:51.043648 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.043300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53772)
0926 21:35:51.043438 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.043516 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 21:35:51.043938 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
I20260926 21:35:51.135265 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } attempt: 3
I20260926 21:35:51.136683 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.136139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53780)
0926 21:35:51.136352 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.136452 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 21:35:51.137928 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.137522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53796)
0926 21:35:51.137702 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.137792 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 21:35:51.138168 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
I20260926 21:35:51.281440 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } attempt: 4
I20260926 21:35:51.282716 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.282137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53806)
0926 21:35:51.282408 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.282559 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 21:35:51.283831 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.283470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53812)
0926 21:35:51.283629 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.283704 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:35:51.284062 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40971: connect: Connection refused (error 111)
W20260926 21:35:51.289964 23239 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:35:51.290364 23239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:51.290551 23239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:51.290680 23239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:51.298216 23239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:51.298326 23239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:51.298417 23239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:51.298517 23239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:51.298606 23239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:51.308794 23239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35973,127.0.97.61:40971,127.0.97.60:46693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:51.310175 23239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:51.311580 23239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:51.323581 23244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:51.323671 23245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:51.323642 23247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:51.325340 23239 server_base.cc:1034] running on GCE node
I20260926 21:35:51.326169 23239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:51.327785 23239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:51.329020 23239 hybrid_clock.cc:648] HybridClock initialized: now 1790458551328958 us; error 32 us; skew 500 ppm
I20260926 21:35:51.329340 23239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:51.332695 23239 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 21:35:51.334216 23239 webserver.cc:460] Webserver started at http://127.0.97.61:34407/ using document root <none> and password file <none>
I20260926 21:35:51.334997 23239 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:51.335086 23239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:51.335356 23239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:51.338572 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/data/instance:
uuid: "780b444480434dad8e45b2c758819f36"
format_stamp: "Formatted at 2026-09-26 21:35:51 on dist-test-slave-87bt"
server_key: "b36eccf7b145adc24c65a625f0c36dbb"
server_key_iv: "e4480e2aea41e32a572d84a1ec7ad51a"
server_key_version: "encryptionkey@0"
I20260926 21:35:51.339376 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/wal/instance:
uuid: "780b444480434dad8e45b2c758819f36"
format_stamp: "Formatted at 2026-09-26 21:35:51 on dist-test-slave-87bt"
server_key: "b36eccf7b145adc24c65a625f0c36dbb"
server_key_iv: "e4480e2aea41e32a572d84a1ec7ad51a"
server_key_version: "encryptionkey@0"
I20260926 21:35:51.344650 23239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:35:51.348300 23254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:51.349756 23239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:35:51.349913 23239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/wal
uuid: "780b444480434dad8e45b2c758819f36"
format_stamp: "Formatted at 2026-09-26 21:35:51 on dist-test-slave-87bt"
server_key: "b36eccf7b145adc24c65a625f0c36dbb"
server_key_iv: "e4480e2aea41e32a572d84a1ec7ad51a"
server_key_version: "encryptionkey@0"
I20260926 21:35:51.350101 23239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:51.366845 23239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:51.371603 23239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:51.371852 23239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:51.390928 23239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40971
I20260926 21:35:51.390957 23305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40971 every 8 connection(s)
I20260926 21:35:51.392374 23239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:35:51.395641 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 23239
I20260926 21:35:51.395828 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-1/wal/instance
I20260926 21:35:51.396340 388 external_mini_cluster.cc:1428] Setting key 9944e6dd9b6f87e8664f8c0fdae94791
I20260926 21:35:51.396747 23306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:51.406901 23306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 21:35:51.428576 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.409579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.0.97.62:35973)
0926 21:35:51.409759 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.409764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.411175 (+ 1411us) server_negotiation.cc:407] Connection header received
0926 21:35:51.413123 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.413129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.413223 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.413339 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.415053 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.415785 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.417455 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.417905 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.418066 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.421813 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.421818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.421822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.421857 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.425053 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.426084 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.426089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.426091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.426213 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.426789 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.426793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.426795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.427074 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.427188 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.428171 (+ 983us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.428353 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:35:51.430457 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.410190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35973 (local address 127.0.0.1:33550)
0926 21:35:51.411000 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.411048 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:51.412822 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:51.413586 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:51.413596 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:51.414153 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:51.414861 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.414880 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.415972 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.415976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.417252 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.417267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.418169 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.418175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.418506 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.419299 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:51.419327 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:51.421544 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:51.425231 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.425239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.425255 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.425933 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.426371 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.426379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.426386 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.426669 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.427211 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:51.427222 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:51.427763 (+ 541us) client_negotiation.cc:769] Sending connection context
0926 21:35:51.428145 (+ 382us) client_negotiation.cc:240] Negotiation successful
0926 21:35:51.428757 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":357,"threads_started":1}
I20260926 21:35:51.432462 23306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:35:51.449826 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.433439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53814)
0926 21:35:51.433690 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.433711 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:51.433842 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:51.435138 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:51.435142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:51.435800 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:51.436045 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.436060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.437789 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.437798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.439453 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.439472 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.440380 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.440388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.440577 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.441246 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:51.441274 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:51.443891 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:51.447150 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.447157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.447164 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.447504 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.447890 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.447895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.447898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.448022 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.448679 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:51.448685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:51.448827 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 21:35:51.448971 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 21:35:51.449370 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 21:35:51.449867 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.433516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.0.97.61:40971)
0926 21:35:51.434585 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.434596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.434629 (+ 33us) server_negotiation.cc:407] Connection header received
0926 21:35:51.434689 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.434698 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.434958 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.435133 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.436289 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.437624 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.439611 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.440197 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.440318 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.444118 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.444131 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.444146 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.444175 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.447008 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.447632 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.447647 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.447649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.447763 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.448117 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.448124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.448128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.448559 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.448779 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.449142 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.449503 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":456,"threads_started":1}
I20260926 21:35:51.452706 23306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:35:51.453889 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.453373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45548)
0926 21:35:51.453552 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.453717 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:35:51.454142 23257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
I20260926 21:35:51.455757 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.455435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45550)
0926 21:35:51.455562 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.455644 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 21:35:51.455981 23306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 21:35:51.461283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
/tmp/dist-test-taskUh_ENg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46693
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35973,127.0.97.61:40971,127.0.97.60:46693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:35:51.505337 23306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 1
I20260926 21:35:51.506592 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.506109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45564)
0926 21:35:51.506300 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.506390 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 21:35:51.507544 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.507184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45576)
0926 21:35:51.507368 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.507441 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 21:35:51.507779 23306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
I20260926 21:35:51.575992 23306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 2
I20260926 21:35:51.577178 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.576749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45592)
0926 21:35:51.576935 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.577033 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 21:35:51.578105 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.577789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45608)
0926 21:35:51.577929 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.577999 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 21:35:51.578327 23306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
I20260926 21:35:51.583328 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } attempt: 5
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:35:51.598467 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.584030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53824)
0926 21:35:51.584262 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.584286 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:51.584451 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:51.584816 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:51.584823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:51.585486 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:51.585724 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.585739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.586739 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.586745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.588105 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.588120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.588858 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.588863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.589066 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.589757 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:51.589781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:51.592312 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:51.596208 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.596212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.596215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.596603 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.596991 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.596994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.596996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.597122 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.597698 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:51.597701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:51.597848 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 21:35:51.597962 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 21:35:51.598165 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 21:35:51.598651 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.584127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.0.97.61:40971)
0926 21:35:51.584281 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.584286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.584368 (+ 82us) server_negotiation.cc:407] Connection header received
0926 21:35:51.584586 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.584590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.584661 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.584740 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.585870 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.586603 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.588274 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.588709 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.588806 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.592532 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.592543 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.592551 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.592600 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.596051 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.596723 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.596729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.596732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.596878 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.597218 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.597222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.597226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.597592 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.597743 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.598095 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.598322 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:35:51.599566 23232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:35:51.600672 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.600213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45620)
0926 21:35:51.600407 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.600500 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 21:35:51.601624 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.601237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45634)
0926 21:35:51.601388 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.601480 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:35:51.601830 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
I20260926 21:35:51.653117 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 1
I20260926 21:35:51.654333 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.653803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45648)
0926 21:35:51.654024 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.654143 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 21:35:51.655408 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.654996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45664)
0926 21:35:51.655157 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.655253 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:35:51.655637 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
I20260926 21:35:51.669539 23306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 3
I20260926 21:35:51.670868 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.670323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45670)
0926 21:35:51.670571 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.670681 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 21:35:51.672058 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.671589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45672)
0926 21:35:51.671873 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.671951 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
W20260926 21:35:51.672308 23306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
I20260926 21:35:51.723903 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 2
I20260926 21:35:51.725266 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.724673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45680)
0926 21:35:51.724911 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.725047 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 21:35:51.726378 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.725952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45692)
0926 21:35:51.726129 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.726219 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 21:35:51.726616 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
W20260926 21:35:51.797394 23313 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:35:51.797786 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:35:51.797923 23313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:35:51.798048 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:35:51.805378 23313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:35:51.805476 23313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:35:51.805599 23313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:35:51.805677 23313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:35:51.805778 23313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:35:51.815507 23306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 4
I20260926 21:35:51.815987 23313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35973,127.0.97.61:40971,127.0.97.60:46693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:35:51.816823 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.816264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45696)
0926 21:35:51.816575 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.816666 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260926 21:35:51.817618 23313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:35:51.817889 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.817513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45706)
0926 21:35:51.817664 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.817765 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:35:51.818079 23306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
I20260926 21:35:51.819171 23313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:35:51.831239 23321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:51.831347 23319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:35:51.831912 23318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:35:51.832114 23313 server_base.cc:1034] running on GCE node
I20260926 21:35:51.832828 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 3
I20260926 21:35:51.832934 23313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:35:51.833878 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.833439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45714)
0926 21:35:51.833641 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.833725 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 21:35:51.834697 23313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:35:51.835053 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.834552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45722)
0926 21:35:51.834789 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.834881 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 21:35:51.835268 23232 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46693: connect: Connection refused (error 111)
I20260926 21:35:51.835949 23313 hybrid_clock.cc:648] HybridClock initialized: now 1790458551835885 us; error 35 us; skew 500 ppm
I20260926 21:35:51.836288 23313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:35:51.839593 23313 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 21:35:51.841002 23313 webserver.cc:460] Webserver started at http://127.0.97.60:38393/ using document root <none> and password file <none>
I20260926 21:35:51.841694 23313 fs_manager.cc:359] Metadata directory not provided
I20260926 21:35:51.841828 23313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:35:51.842131 23313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:35:51.845094 23313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/data/instance:
uuid: "4bd575f9bb064d2cb17b21e83369d991"
format_stamp: "Formatted at 2026-09-26 21:35:51 on dist-test-slave-87bt"
server_key: "3915b3ed3c358f59a006fb379a8550ac"
server_key_iv: "06be37628ec7d9c737f4b6fcd4bf4726"
server_key_version: "encryptionkey@0"
I20260926 21:35:51.845886 23313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/wal/instance:
uuid: "4bd575f9bb064d2cb17b21e83369d991"
format_stamp: "Formatted at 2026-09-26 21:35:51 on dist-test-slave-87bt"
server_key: "3915b3ed3c358f59a006fb379a8550ac"
server_key_iv: "06be37628ec7d9c737f4b6fcd4bf4726"
server_key_version: "encryptionkey@0"
I20260926 21:35:51.851049 23313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 21:35:51.854360 23328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:35:51.855751 23313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:35:51.855922 23313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/wal
uuid: "4bd575f9bb064d2cb17b21e83369d991"
format_stamp: "Formatted at 2026-09-26 21:35:51 on dist-test-slave-87bt"
server_key: "3915b3ed3c358f59a006fb379a8550ac"
server_key_iv: "06be37628ec7d9c737f4b6fcd4bf4726"
server_key_version: "encryptionkey@0"
I20260926 21:35:51.856127 23313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:35:51.873004 23313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:35:51.877977 23313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:35:51.878245 23313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:35:51.897300 23313 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46693
I20260926 21:35:51.897324 23379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46693 every 8 connection(s)
I20260926 21:35:51.898681 23313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:35:51.902356 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:35:51.905728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu as pid 23313
I20260926 21:35:51.905913 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUh_ENg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458187148965-388-0/minicluster-data/master-2/wal/instance
I20260926 21:35:51.906456 388 external_mini_cluster.cc:1428] Setting key 133f99c7161fa5738a2cd11db0af7a86
I20260926 21:35:51.909866 23380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:51.928822 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.913309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.0.97.62:35973)
0926 21:35:51.913510 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.913517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.914082 (+ 565us) server_negotiation.cc:407] Connection header received
0926 21:35:51.914233 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.914238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.914304 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.914390 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.915932 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.917051 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.918903 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.919545 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.919689 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.923148 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.923154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.923157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.923193 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.925792 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.926528 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.926535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.926538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.926707 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.927139 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.927143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.927145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.927415 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.927528 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.928189 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.928431 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:35:51.929927 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.913203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35973 (local address 127.0.0.1:33552)
0926 21:35:51.913900 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.913955 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:51.914111 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:51.914493 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:51.914504 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:51.915332 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:51.915679 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.915689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.917205 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.917209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.918758 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.918770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.919758 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.919763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.919886 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.920458 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:51.920480 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:51.922916 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:51.925933 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.925945 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.925959 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.926348 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.926828 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.926833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.926836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.926994 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.927537 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:51.927545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:51.927743 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 21:35:51.928024 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 21:35:51.928601 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":121,"thread_start_us":307,"threads_started":1}
I20260926 21:35:51.929946 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.908328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.0.97.60:46693)
0926 21:35:51.909130 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.909142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.909186 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:35:51.910943 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.910966 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.911394 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.911804 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.913643 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.915690 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.917466 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.918488 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.918680 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.922981 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.922996 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.923017 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.923059 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.925841 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.926561 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.926567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.926570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.926704 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.927069 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.927075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.927078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.927614 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.927926 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.928121 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.928485 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":593,"thread_start_us":376,"threads_started":1}
I20260926 21:35:51.931845 23380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:35:51.946107 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.932602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40971 (local address 127.0.0.1:53838)
0926 21:35:51.932890 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.932909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:51.933023 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:51.933426 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:51.933429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:51.933906 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:51.934109 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.934120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.935371 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.935375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.936584 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.936600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.937283 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.937292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.937514 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.938074 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:51.938101 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:51.940601 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:51.943763 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.943770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.943776 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.944302 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.944659 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.944664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.944667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.944838 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.945304 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:51.945311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:51.945475 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 21:35:51.945629 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 21:35:51.945868 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260926 21:35:51.946388 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.932600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.0.97.61:40971)
0926 21:35:51.932776 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.932785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.932977 (+ 192us) server_negotiation.cc:407] Connection header received
0926 21:35:51.933195 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.933201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.933290 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.933397 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.934283 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.935203 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.936723 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.937137 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.937280 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.940804 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.940814 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.940819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.940868 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.943628 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.944414 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.944418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.944421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.944542 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.944941 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.944943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.944945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.945177 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.945294 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.945739 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.945962 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:35:51.947463 23380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:35:51.948292 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.933039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.0.97.62:35973)
0926 21:35:51.933192 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.933197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.933222 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:35:51.933343 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.933348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.933418 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.933506 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.934904 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.935757 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.937390 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.938036 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.938185 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.942541 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.942546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.942549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.942589 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.945484 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.946285 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.946290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.946293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.946494 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.946882 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.946886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.946888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.947240 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.947505 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.947736 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.947952 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:35:51 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:35:51.961124 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.948169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45742)
0926 21:35:51.948329 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:51.948348 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:51.948450 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:51.949036 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:51.949040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:51.949428 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:51.949560 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.949569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.950662 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.950670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.951490 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:51.951500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.952135 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.952140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:51.952342 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.952880 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:51.952898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:51.954993 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:51.958475 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.958482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.958486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.958995 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.959412 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:51.959418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:51.959423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:51.959567 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:51.960163 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:51.960169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:51.960356 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 21:35:51.960480 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 21:35:51.960776 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 21:35:51.961406 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.948601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.0.97.60:46693)
0926 21:35:51.948781 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.948786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.948803 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:35:51.948850 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.948853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.948902 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.948973 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.949686 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.950554 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.951616 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.952008 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.952130 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.955234 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.955241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.955245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.955283 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.958324 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.959101 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.959105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.959107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.959244 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.959679 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.959684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.959687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.960043 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.960183 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.960683 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.960933 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":52}
I20260926 21:35:51.965543 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:51.949980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.0.97.61:40971)
0926 21:35:51.950145 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:51.950153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:51.950171 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:35:51.950217 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:51.950221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:51.950301 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:51.950399 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:51.951547 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.952250 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.953721 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:51.954188 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:51.954353 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:51.959401 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:51.959407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:51.959412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:51.959454 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:51.963122 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.963721 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.963727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.963730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.963874 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:51.964197 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:51.964201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:51.964204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:51.964540 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:51.964816 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:51.964990 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 21:35:51.965214 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 21:35:51.967535 23380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: Bootstrap starting.
I20260926 21:35:51.970310 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:35:51.971607 23380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:51.972923 23380 log.cc:824] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:51.975687 23380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: No bootstrap required, opened a new log
I20260926 21:35:51.973387 23207 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0af5008a18fc4aca9139489044346f11"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33560
I20260926 21:35:51.982910 23380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } }
I20260926 21:35:51.983258 23380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd575f9bb064d2cb17b21e83369d991, State: Initialized, Role: FOLLOWER
I20260926 21:35:51.983709 23380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } }
I20260926 21:35:51.984601 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:51.984921 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:51.985939 23380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:35:51.995500 23397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:35:51.995627 23397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:35:51.993580 23313 master_runner.cc:186] Error getting master registration for 127.0.97.62:35973: OK, instance_id { permanent_uuid: "0af5008a18fc4aca9139489044346f11" instance_seqno: 1790458550868788 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:35:52.004984 23313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:35:52.012460 23232 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 4
Sep 26 21:35:52 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:35:52.029567 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:52.013242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45746)
0926 21:35:52.013465 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:52.013486 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:52.013615 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:52.013968 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:52.013973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:52.014512 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:52.014744 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:52.014756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.015813 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.015819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:52.017322 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:52.017341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.018402 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.018411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:52.018658 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:52.019362 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:52.019386 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:52.022153 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:52.026452 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:52.026459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:52.026463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:52.027166 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:52.027621 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:52.027626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:52.027629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:52.027805 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:52.028519 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:52.028524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:52.028704 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 21:35:52.028843 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 21:35:52.029163 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 21:35:52.029616 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:52.013274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.0.97.60:46693)
0926 21:35:52.013521 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.013529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.013552 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:35:52.013746 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.013752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.013829 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.013939 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.014894 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.015664 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.017513 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.018232 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.018375 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:52.022371 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:52.022378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:52.022383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:52.022444 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:52.026233 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.027332 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.027339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.027343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.027492 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.027930 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.027935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.027939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.028398 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:52.028566 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:52.028989 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 21:35:52.029235 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 21:35:52.035722 23232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11: Bootstrap starting.
I20260926 21:35:52.039937 23232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:52.041299 23232 log.cc:824] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:52.044149 23232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11: No bootstrap required, opened a new log
I20260926 21:35:52.049327 23232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } }
I20260926 21:35:52.049680 23232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af5008a18fc4aca9139489044346f11, State: Initialized, Role: FOLLOWER
I20260926 21:35:52.050143 23232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } }
I20260926 21:35:52.051148 23400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.051537 23400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:52.052320 23232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:52.052415 23207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 21:35:52.052652 23207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:52.056212 23207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } }
I20260926 21:35:52.058835 23355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0af5008a18fc4aca9139489044346f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4bd575f9bb064d2cb17b21e83369d991"
I20260926 21:35:52.059052 23207 leader_election.cc:288] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [CANDIDATE]: Term 1 election: Requested vote from peers 780b444480434dad8e45b2c758819f36 (127.0.97.61:40971), 4bd575f9bb064d2cb17b21e83369d991 (127.0.97.60:46693)
I20260926 21:35:52.059358 23355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:52.063095 23355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0af5008a18fc4aca9139489044346f11 in term 1.
I20260926 21:35:52.063858 23183 leader_election.cc:302] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af5008a18fc4aca9139489044346f11, 4bd575f9bb064d2cb17b21e83369d991; no voters:
I20260926 21:35:52.064389 23400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:35:52.075820 23400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [term 1 LEADER]: Becoming Leader. State: Replica: 0af5008a18fc4aca9139489044346f11, State: Running, Role: LEADER
I20260926 21:35:52.076489 23400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } }
I20260926 21:35:52.082832 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af5008a18fc4aca9139489044346f11. Latest consensus state: current_term: 1 leader_uuid: "0af5008a18fc4aca9139489044346f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.083280 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:52.087734 23415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:35:52.102178 23415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 21:35:52.108021 23186 tablet.cc:2369] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:35:52.117252 23306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } attempt: 5
W20260926 21:35:52.130570 23334 tablet.cc:2369] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:35:52.119432 23165 master_runner.cc:186] Error getting master registration for 127.0.97.61:40971: OK, instance_id { permanent_uuid: "780b444480434dad8e45b2c758819f36" instance_seqno: 1790458551385393 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:35:52.158658 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:52.094565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.0.97.61:40971)
0926 21:35:52.094769 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.094776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.106997 (+ 12221us) server_negotiation.cc:407] Connection header received
0926 21:35:52.110603 (+ 3606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.110613 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.110733 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.110828 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.113293 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.114573 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.116725 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.117340 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.117454 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:52.144951 (+ 27497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:52.144960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:52.144966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:52.145015 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:52.148756 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.154478 (+ 5722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.154487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.154492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.154737 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.155936 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.155942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.155945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.156343 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:52.156582 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:52.157894 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0926 21:35:52.158222 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 21:35:52.160343 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:52.107172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.97.60:46693)
0926 21:35:52.107377 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.107386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.112156 (+ 4770us) server_negotiation.cc:407] Connection header received
0926 21:35:52.112602 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.112609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.112715 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.112788 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.117722 (+ 4934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.118782 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.131711 (+ 12929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.132428 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.132548 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:52.153571 (+ 21023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:52.153578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:52.153584 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:52.153634 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:52.157149 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.157969 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.157974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.157977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.158134 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.158582 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.158587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.158591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.158930 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:52.159167 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:52.159773 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 21:35:52.159992 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:35:52.163360 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:52.094714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.0.97.62:35973)
0926 21:35:52.094896 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.094904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.106445 (+ 11541us) server_negotiation.cc:407] Connection header received
0926 21:35:52.107614 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.107623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.107723 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.107835 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.133798 (+ 25963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.135163 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.140890 (+ 5727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.141602 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.141708 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:52.149803 (+ 8095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:52.149812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:52.149818 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:52.149874 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:52.154003 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.157145 (+ 3142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.157151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.157155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.157340 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.161132 (+ 3792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.161138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.161142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.161623 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:52.161923 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:52.162667 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 21:35:52.162974 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:35:52.165678 23165 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 21:35:52 dist-test-slave-87bt krb5kdc[23152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:35:52.174368 23355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [term 1 FOLLOWER]: Refusing update from remote peer 0af5008a18fc4aca9139489044346f11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:52.175442 23410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:52.184564 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:52.117873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46693 (local address 127.0.0.1:45758)
0926 21:35:52.118181 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 21:35:52.118201 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:35:52.118311 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:35:52.150608 (+ 32297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:35:52.150615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:35:52.151168 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:35:52.151417 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:52.151434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.162922 (+ 11488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.162928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:52.164366 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:35:52.164381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.165329 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.165336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:35:52.165530 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:52.166192 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:35:52.166214 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:35:52.168938 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:35:52.180626 (+ 11688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:52.180635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:52.180643 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:52.181513 (+ 870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:52.182040 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:35:52.182048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:35:52.182053 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:35:52.182274 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:35:52.183176 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:35:52.183184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:35:52.183377 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 21:35:52.183558 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 21:35:52.183943 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260926 21:35:52.185669 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:52.119064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.0.97.60:46693)
0926 21:35:52.141020 (+ 21956us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.141025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.141046 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:35:52.141113 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.141118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.141203 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.151613 (+ 10410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.151917 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.153264 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.164562 (+ 11298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:52.165171 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:52.166337 (+ 1166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:52.169753 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:35:52.169760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:35:52.169766 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:35:52.169801 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:35:52.180443 (+ 10642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.181686 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.181692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.181696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.181892 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:35:52.182451 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:35:52.182459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:35:52.182465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:35:52.182924 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:35:52.183081 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:52.185030 (+ 1949us) server_negotiation.cc:299] Negotiation successful
0926 21:35:52.185274 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21804,"thread_start_us":22215,"threads_started":1}
I20260926 21:35:52.192395 23306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36: Bootstrap starting.
I20260926 21:35:52.196893 23306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36: Neither blocks nor log segments found. Creating new log.
I20260926 21:35:52.197618 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af5008a18fc4aca9139489044346f11. Latest consensus state: current_term: 1 leader_uuid: "0af5008a18fc4aca9139489044346f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.197952 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:52.198561 23306 log.cc:824] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36: Log is configured to *not* fsync() on all Append() calls
I20260926 21:35:52.202231 23423 mvcc.cc:205] Tried to move back new op lower bound from 7333718229692944384 to 7333718229311492096. Current Snapshot: MvccSnapshot[applied={T|T < 7333718229692944384}]
I20260926 21:35:52.205021 23425 mvcc.cc:205] Tried to move back new op lower bound from 7333718229692944384 to 7333718229311492096. Current Snapshot: MvccSnapshot[applied={T|T < 7333718229692944384}]
I20260926 21:35:52.208565 23306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36: No bootstrap required, opened a new log
I20260926 21:35:52.216138 23400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0af5008a18fc4aca9139489044346f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.216480 23400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: This master's current role is: LEADER
I20260926 21:35:52.219327 23415 catalog_manager.cc:1347] Generated new cluster ID: 8194a9f79b6c4bf2a48ce42182677e5d
I20260926 21:35:52.219450 23415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:35:52.220250 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0af5008a18fc4aca9139489044346f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.220559 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:52.238734 23415 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:35:52.240651 23415 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:35:52.236899 23306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } }
I20260926 21:35:52.249902 23306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780b444480434dad8e45b2c758819f36, State: Initialized, Role: FOLLOWER
I20260926 21:35:52.250589 23306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } }
I20260926 21:35:52.255726 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.258338 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:52.258213 23306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:35:52.259512 23281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0af5008a18fc4aca9139489044346f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "780b444480434dad8e45b2c758819f36"
I20260926 21:35:52.261409 23280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:35:52.265341 23280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [term 1 FOLLOWER]: Refusing update from remote peer 0af5008a18fc4aca9139489044346f11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:35:52.266383 23400 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:35:52.277688 23415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11: Generated new TSK 0
I20260926 21:35:52.278872 23415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:35:52.350682 23281 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0af5008a18fc4aca9139489044346f11 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260926 21:35:52.352510 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af5008a18fc4aca9139489044346f11. Latest consensus state: current_term: 1 leader_uuid: "0af5008a18fc4aca9139489044346f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.352855 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:52.353386 23447 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 21:35:52.353308 23440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0af5008a18fc4aca9139489044346f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.353657 23440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:35:52.353523 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0af5008a18fc4aca9139489044346f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af5008a18fc4aca9139489044346f11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35973 } } peers { permanent_uuid: "780b444480434dad8e45b2c758819f36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40971 } } peers { permanent_uuid: "4bd575f9bb064d2cb17b21e83369d991" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46693 } } }
I20260926 21:35:52.353813 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af5008a18fc4aca9139489044346f11 [sys.catalog]: This master's current role is: LEADER
W20260926 21:35:52.393639 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.381150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.0.97.61:40971)
0926 21:35:52.381340 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.381347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.391018 (+ 9671us) server_negotiation.cc:407] Connection header received
0926 21:35:52.391547 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.391556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.391691 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.391796 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.393338 (+ 1542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53864: BlockingRecv error: recv got EOF from 127.0.0.1:53864 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 21:35:52.394959 23235 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.381221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.0.97.62:35973)
0926 21:35:52.381419 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.381425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.392771 (+ 11346us) server_negotiation.cc:407] Connection header received
0926 21:35:52.393263 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.393271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.393359 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.393433 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.394737 (+ 1304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33590: BlockingRecv error: recv got EOF from 127.0.0.1:33590 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 21:35:52.396188 23424 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.391202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.0.97.60:46693)
0926 21:35:52.391384 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.391390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.394469 (+ 3079us) server_negotiation.cc:407] Connection header received
0926 21:35:52.394747 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.394757 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.394880 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.394967 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.395963 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45766: BlockingRecv error: recv got EOF from 127.0.0.1:45766 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 21:35:52.430213 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.427750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.0.97.61:40971)
0926 21:35:52.427945 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.427952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.428532 (+ 580us) server_negotiation.cc:407] Connection header received
0926 21:35:52.428800 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.428806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.428953 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.429067 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.429930 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53876: BlockingRecv error: recv got EOF from 127.0.0.1:53876 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 21:35:52.438779 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.436459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.0.97.61:40971)
0926 21:35:52.436649 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.436655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.437163 (+ 508us) server_negotiation.cc:407] Connection header received
0926 21:35:52.437520 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.437526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.437624 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.437740 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.438567 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53888: BlockingRecv error: recv got EOF from 127.0.0.1:53888 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 21:35:52.448661 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.444631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.0.97.61:40971)
0926 21:35:52.444838 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.444847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.447079 (+ 2232us) server_negotiation.cc:407] Connection header received
0926 21:35:52.447298 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.447304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.447423 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.447555 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.448422 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53896: BlockingRecv error: recv got EOF from 127.0.0.1:53896 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 21:35:52.458901 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.456641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.0.97.61:40971)
0926 21:35:52.456863 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.456870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.457408 (+ 538us) server_negotiation.cc:407] Connection header received
0926 21:35:52.457619 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.457626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.457720 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.457833 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.458677 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53906: BlockingRecv error: recv got EOF from 127.0.0.1:53906 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260926 21:35:52.473333 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.470749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.97.61:40971)
0926 21:35:52.470962 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.470968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.471511 (+ 543us) server_negotiation.cc:407] Connection header received
0926 21:35:52.471755 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.471762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.471868 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.471980 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.473088 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53908: BlockingRecv error: recv got EOF from 127.0.0.1:53908 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 21:35:52.495985 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.493599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.0.97.61:40971)
0926 21:35:52.493786 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.493795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.494374 (+ 579us) server_negotiation.cc:407] Connection header received
0926 21:35:52.494617 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.494623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.494724 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.494849 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.495738 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53910: BlockingRecv error: recv got EOF from 127.0.0.1:53910 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 21:35:52.540720 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.538114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.0.97.61:40971)
0926 21:35:52.538325 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.538333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.538993 (+ 660us) server_negotiation.cc:407] Connection header received
0926 21:35:52.539255 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.539262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.539379 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.539508 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.540435 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53912: BlockingRecv error: recv got EOF from 127.0.0.1:53912 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 21:35:52.610400 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.608446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.0.97.61:40971)
0926 21:35:52.608670 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.608679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.609039 (+ 360us) server_negotiation.cc:407] Connection header received
0926 21:35:52.609264 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.609268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.609339 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.609446 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.610163 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53920: BlockingRecv error: recv got EOF from 127.0.0.1:53920 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 21:35:52.743614 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:52.741667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.0.97.61:40971)
0926 21:35:52.741905 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:52.741912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:52.742166 (+ 254us) server_negotiation.cc:407] Connection header received
0926 21:35:52.742409 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:52.742414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:52.742535 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:52.742640 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:52.743326 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53924: BlockingRecv error: recv got EOF from 127.0.0.1:53924 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:35:52.997956 23397 catalog_manager.cc:1259] Loaded cluster ID: 8194a9f79b6c4bf2a48ce42182677e5d
I20260926 21:35:52.998121 23397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: loading cluster ID for follower catalog manager: success
I20260926 21:35:53.001534 23397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: acquiring CA information for follower catalog manager: success
I20260926 21:35:53.005204 23397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4bd575f9bb064d2cb17b21e83369d991: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 21:35:53.009060 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:35:53.007243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.0.97.61:40971)
0926 21:35:53.007487 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:53.007494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:53.007664 (+ 170us) server_negotiation.cc:407] Connection header received
0926 21:35:53.007906 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:53.007910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:53.007983 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:53.008099 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:35:53.008829 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53936: BlockingRecv error: recv got EOF from 127.0.0.1:53936 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 21:35:53.357614 23447 catalog_manager.cc:1259] Loaded cluster ID: 8194a9f79b6c4bf2a48ce42182677e5d
I20260926 21:35:53.357736 23447 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36: loading cluster ID for follower catalog manager: success
I20260926 21:35:53.360713 23447 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36: acquiring CA information for follower catalog manager: success
I20260926 21:35:53.362778 23447 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 780b444480434dad8e45b2c758819f36: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:35:53.531173 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:53.524642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.0.97.61:40971)
0926 21:35:53.525460 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:53.525465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:53.525480 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:35:53.525541 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:53.525544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:53.525692 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:53.525835 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:53.526277 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.527283 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.528712 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.529177 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.529341 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:53.529866 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:53.530502 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:53.530677 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:53.530774 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 21:35:53.530865 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":481,"threads_started":1}
I20260926 21:35:53.544911 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:53.538062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.0.97.60:46693)
0926 21:35:53.538587 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:53.538594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:53.538953 (+ 359us) server_negotiation.cc:407] Connection header received
0926 21:35:53.539157 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:53.539163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:53.539368 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:53.539557 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:53.539952 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.541019 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.542331 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.542965 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.543098 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:53.543585 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:53.544282 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:53.544433 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:53.544495 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 21:35:53.544593 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":264,"threads_started":1}
I20260926 21:35:53.572384 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:53.551455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.0.97.62:35973)
0926 21:35:53.563629 (+ 12174us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:53.563637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:53.564220 (+ 583us) server_negotiation.cc:407] Connection header received
0926 21:35:53.564347 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:53.564353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:53.564608 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:53.565577 (+ 969us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:53.565814 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.567235 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.569384 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.570010 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.570755 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:53.570938 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:53.571366 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:53.571913 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:53.571967 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 21:35:53.572052 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12023,"thread_start_us":11913,"threads_started":1}
I20260926 21:35:53.574913 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:53.564302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.0.97.60:46693)
0926 21:35:53.564484 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:53.564490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:53.567970 (+ 3480us) server_negotiation.cc:407] Connection header received
0926 21:35:53.568259 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:53.568266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:53.568358 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:53.568420 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:53.569228 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.570308 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.572572 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.573192 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.573292 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:53.574023 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:53.574197 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:53.574263 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:53.574516 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 21:35:53.574601 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:35:53.575671 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:35:53.551276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.0.97.61:40971)
0926 21:35:53.551437 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 21:35:53.551444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:35:53.567612 (+ 16168us) server_negotiation.cc:407] Connection header received
0926 21:35:53.568019 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:35:53.568024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:35:53.568118 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:35:53.568195 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:35:53.568988 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.570104 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.572374 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:35:53.572964 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:35:53.573063 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:35:53.574600 (+ 1537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:35:53.574728 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:35:53.574807 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 21:35:53.575293 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 21:35:53.575371 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 21:35:53.577076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 23165
I20260926 21:35:53.578266 23227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:53.688694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 23165
I20260926 21:35:53.709231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 23239
I20260926 21:35:53.710397 23301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:53.819504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 23239
I20260926 21:35:53.839569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu pid 23313
I20260926 21:35:53.840669 23375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:35:53.946374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUh_ENg/build/asan/bin/kudu with pid 23313
2026-09-26T21:35:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4914 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9299 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366760 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST